2024-09-22T02:47:47.632 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-22T02:47:47.651 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T02:47:47.745 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914712 branch: squid description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914712' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client.0: admin socket: /var/run/ceph/ceph-$name.asok global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: squid env: CRIMSON_COMPAT: '1' sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 1158 sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBC7zrS1ffvustjUUkTcbVOw2gZT7UUiNJzcSUenkt3NmUxvIgPkBgAlZ6j97Cw8EPvxVDHNzjmjy6LDCX3auKA= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2evev0l2v39HqdhVel1zzTPuwFmZzH+X42VuM4ETyQs1pX/AYtv05/u2TpbJogC5QzSHBF8V/wVvU5UL8NWrM= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - radosbench: clients: - client.0 time: 150 - admin_socket: client.0: objecter_requests: test: http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch} teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_21:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-22T02:47:47.745 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-22T02:47:47.746 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-22T02:47:47.746 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-22T02:47:47.747 INFO:teuthology.task.internal:Checking packages... 2024-09-22T02:47:47.767 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '4e45e6b490e6cbe082cdf155fea780438ec53fc7' 2024-09-22T02:47:47.767 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-22T02:47:47.767 INFO:teuthology.packaging:ref: None 2024-09-22T02:47:47.767 INFO:teuthology.packaging:tag: None 2024-09-22T02:47:47.767 INFO:teuthology.packaging:branch: squid 2024-09-22T02:47:47.767 INFO:teuthology.packaging:sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T02:47:47.767 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-22T02:47:47.954 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373.g4e45e6b4 2024-09-22T02:47:47.956 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-22T02:47:47.963 INFO:teuthology.task.internal:no buildpackages task found 2024-09-22T02:47:47.963 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-22T02:47:47.969 INFO:teuthology.task.internal:Saving configuration 2024-09-22T02:47:47.984 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-22T02:47:47.990 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-22T02:47:48.017 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914712', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 02:42:52.892637', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBC7zrS1ffvustjUUkTcbVOw2gZT7UUiNJzcSUenkt3NmUxvIgPkBgAlZ6j97Cw8EPvxVDHNzjmjy6LDCX3auKA='} 2024-09-22T02:47:48.037 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914712', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 02:42:52.891480', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2evev0l2v39HqdhVel1zzTPuwFmZzH+X42VuM4ETyQs1pX/AYtv05/u2TpbJogC5QzSHBF8V/wVvU5UL8NWrM='} 2024-09-22T02:47:48.038 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-22T02:47:48.044 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-09-22T02:47:48.044 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-22T02:47:48.044 INFO:teuthology.run_tasks:Running task console_log... 2024-09-22T02:47:48.104 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb7a8734160>, signals=[15]) 2024-09-22T02:47:48.104 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-22T02:47:48.110 INFO:teuthology.task.internal:Opening connections... 2024-09-22T02:47:48.110 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-09-22T02:47:48.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:47:48.185 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-09-22T02:47:48.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:47:48.261 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-22T02:47:48.273 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-09-22T02:47:48.291 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-09-22T02:47:48.291 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-09-22T02:47:48.348 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2024-09-22T02:47:48.348 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="9" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="9" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el9" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:LOGO="fedora-logo-icon" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T02:47:48.349 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T02:47:48.350 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-09-22T02:47:48.369 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-09-22T02:47:48.387 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-09-22T02:47:48.388 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-09-22T02:47:48.445 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-09-22T02:47:48.446 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-09-22T02:47:48.446 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-09-22T02:47:48.446 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-09-22T02:47:48.446 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-09-22T02:47:48.446 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-09-22T02:47:48.446 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T02:47:48.446 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-09-22T02:47:48.446 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-09-22T02:47:48.446 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T02:47:48.446 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-09-22T02:47:48.446 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T02:47:48.447 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T02:47:48.447 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T02:47:48.447 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-09-22T02:47:48.468 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-22T02:47:48.478 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-22T02:47:48.504 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-22T02:47:48.504 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-09-22T02:47:48.507 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-09-22T02:47:48.524 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-22T02:47:48.530 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-22T02:47:48.530 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-09-22T02:47:48.564 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-09-22T02:47:48.598 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-22T02:47:48.653 INFO:teuthology.run_tasks:Running task kernel... 2024-09-22T02:47:48.667 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-22T02:47:48.667 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-22T02:47:48.667 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T02:47:48.668 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T02:47:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T02:47:48.685 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-09-22T02:47:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T02:47:48.686 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-22T02:47:48.743 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-503.el9.x86_64 2024-09-22T02:47:48.743 INFO:teuthology.task.kernel:Running kernel on smithi081: 5.14.0-503.el9.x86_64 2024-09-22T02:47:48.744 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum install -y kernel 2024-09-22T02:47:48.745 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-503.el9.x86_64 2024-09-22T02:47:48.745 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.14.0-503.el9.x86_64 2024-09-22T02:47:48.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-09-22T02:47:51.034 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.3 MB 00:00 2024-09-22T02:47:51.875 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.3 MB 00:01 2024-09-22T02:47:53.010 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-22T02:47:53.876 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-22T02:47:57.227 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-22T02:47:58.169 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-22T02:47:58.895 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 19 kB 00:00 2024-09-22T02:48:00.019 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 19 kB 00:00 2024-09-22T02:48:00.124 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-22T02:48:01.140 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-22T02:48:05.705 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-22T02:48:06.613 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-09-22T02:48:07.552 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T02:48:07.553 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T02:48:07.643 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:48:07.655 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:48:07.655 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-09-22T02:48:07.655 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:48:07.655 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:48:07.655 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T02:48:07.655 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-22T02:48:07.656 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T02:48:07.656 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T02:48:07.656 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T02:48:07.656 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:48:07.656 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:48:07.656 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:48:07.656 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-09-22T02:48:07.656 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:48:07.657 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 79 M 2024-09-22T02:48:07.657 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 126 M 2024-09-22T02:48:07.657 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:48:07.814 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 321 kB/s | 36 kB 00:00 2024-09-22T02:48:08.131 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-09-22T02:48:08.486 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T02:48:08.486 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T02:48:08.573 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 40 MB/s | 35 MB 00:00 2024-09-22T02:48:08.587 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:48:08.598 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:48:08.598 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout:Install 4 Packages 2024-09-22T02:48:08.599 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:48:08.600 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 79 M 2024-09-22T02:48:08.600 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 126 M 2024-09-22T02:48:08.600 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:48:08.764 INFO:teuthology.orchestra.run.smithi081.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 301 kB/s | 36 kB 00:00 2024-09-22T02:48:08.882 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 27 MB/s | 29 MB 00:01 2024-09-22T02:48:08.882 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:48:08.882 INFO:teuthology.orchestra.run.smithi190.stdout:Total 64 MB/s | 79 MB 00:01 2024-09-22T02:48:09.498 INFO:teuthology.orchestra.run.smithi081.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-09-22T02:48:09.561 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:48:09.904 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:48:09.904 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:48:10.006 INFO:teuthology.orchestra.run.smithi081.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 23 MB/s | 29 MB 00:01 2024-09-22T02:48:10.340 INFO:teuthology.orchestra.run.smithi081.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 20 MB/s | 35 MB 00:01 2024-09-22T02:48:10.340 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:48:10.340 INFO:teuthology.orchestra.run.smithi081.stdout:Total 45 MB/s | 79 MB 00:01 2024-09-22T02:48:10.891 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:48:10.891 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:48:11.031 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:48:11.396 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:48:11.396 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:48:12.357 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:48:12.358 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:48:13.367 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:48:14.241 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T02:48:14.340 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:48:14.774 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:48:15.545 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T02:48:15.624 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:48:17.091 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:48:17.218 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:48:18.396 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:48:18.521 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:48:22.067 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:48:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:48:23.336 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:48:23.463 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:48:27.015 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:48:28.324 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:49:05.268 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:49:05.309 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:49:05.599 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:49:05.599 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T02:49:05.599 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:49:05.599 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:49:06.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:49:06.454 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:49:06.454 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:49:06.454 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T02:49:06.454 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T02:49:06.454 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T02:49:06.454 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T02:49:06.454 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:49:06.454 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:49:06.456 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:49:06.496 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:49:06.810 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T02:49:06.841 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:49:06.841 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T02:49:06.841 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:49:06.841 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:49:06.898 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T02:49:06.898 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T02:49:06.899 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T02:49:06.899 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T02:49:06.899 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T02:49:06.899 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-22T02:49:06.899 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T02:49:06.899 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T02:49:06.958 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T02:49:06.959 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T02:49:06.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2024-09-22T02:49:06.990 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2024-09-22T02:49:06.990 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.06 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 82.el9 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Group : Unspecified 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 8274670 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : builder@centos.org 2024-09-22T02:49:06.991 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2024-09-22T02:49:06.992 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T02:49:06.992 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2024-09-22T02:49:06.992 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2024-09-22T02:49:06.992 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:49:06.992 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T02:49:06.992 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T02:49:06.992 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T02:49:06.992 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2024-09-22T02:49:06.992 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:49:06.992 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T02:49:06.993 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T02:49:06.993 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 5.14.0-508.el9.x86_64 2024-09-22T02:49:06.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T02:49:07.637 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-09-22T02:49:07.735 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:49:07.736 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:49:07.736 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:49:07.736 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T02:49:07.736 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T02:49:07.736 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T02:49:07.736 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T02:49:07.736 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:49:07.736 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:49:08.077 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T02:49:08.157 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T02:49:08.157 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T02:49:08.158 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T02:49:08.158 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T02:49:08.158 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T02:49:08.158 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-22T02:49:08.158 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T02:49:08.158 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T02:49:08.216 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T02:49:08.216 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T02:49:08.217 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2024-09-22T02:49:08.254 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2024-09-22T02:49:08.254 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2024-09-22T02:49:08.254 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.06 2024-09-22T02:49:08.254 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 82.el9 2024-09-22T02:49:08.254 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2024-09-22T02:49:08.254 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T02:49:08.254 INFO:teuthology.orchestra.run.smithi081.stdout:Group : Unspecified 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 8274670 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : builder@centos.org 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:49:08.255 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T02:49:08.256 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T02:49:08.256 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T02:49:08.256 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2024-09-22T02:49:08.256 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:49:08.256 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T02:49:08.257 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T02:49:08.257 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 5.14.0-508.el9.x86_64 2024-09-22T02:49:08.257 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T02:49:08.903 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-09-22T02:49:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T02:49:08.940 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-09-22T02:49:08.943 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-22T02:49:08.958 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.mIgkOttmln 2024-09-22T02:49:08.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mIgkOttmln 2024-09-22T02:49:09.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.mIgkOttmln 2024-09-22T02:49:09.172 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.mIgkOttmln is 6KB 2024-09-22T02:49:09.222 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.mIgkOttmln 2024-09-22T02:49:09.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T02:49:09.301 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T02:49:09.301 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T02:49:09.301 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T02:49:09.301 INFO:teuthology.orchestra.run.smithi190.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T02:49:09.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T02:49:09.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-09-22T02:49:10.220 INFO:teuthology.orchestra.run.smithi081.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T02:49:10.256 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-09-22T02:49:10.262 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-09-22T02:49:10.277 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.hJFzzeblqh 2024-09-22T02:49:10.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hJFzzeblqh 2024-09-22T02:49:10.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.hJFzzeblqh 2024-09-22T02:49:10.482 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.hJFzzeblqh is 6KB 2024-09-22T02:49:10.530 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.hJFzzeblqh 2024-09-22T02:49:10.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T02:49:10.608 INFO:teuthology.orchestra.run.smithi081.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T02:49:10.608 INFO:teuthology.orchestra.run.smithi081.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T02:49:10.608 INFO:teuthology.orchestra.run.smithi081.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T02:49:10.608 INFO:teuthology.orchestra.run.smithi081.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T02:49:10.609 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T02:49:10.758 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2024-09-22T02:49:39.490 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-22T02:49:39.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-22T02:49:39.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:49:40.762 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-22T02:49:40.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-22T02:49:40.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:50:15.407 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-09-22T02:50:24.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-22T02:50:24.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:50:39.507 DEBUG:teuthology.orchestra.remote:timed out 2024-09-22T02:50:42.799 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-09-22T02:50:48.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-22T02:50:48.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:50:48.914 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-22T02:50:49.293 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-22T02:50:49.293 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-508.el9.x86_64"... 2024-09-22T02:50:49.293 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-22T02:50:49.308 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-508.el9.x86_64 2024-09-22T02:50:49.308 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T02:50:49.308 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T02:50:49.308 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T02:50:50.309 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-22T02:50:50.310 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T02:50:50.405 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-09-22T02:50:50.432 DEBUG:teuthology.parallel:result is None 2024-09-22T02:50:54.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-22T02:50:54.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:50:57.871 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-09-22T02:51:12.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-22T02:51:12.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:51:13.248 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-09-22T02:51:13.614 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-22T02:51:13.614 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-22T02:51:13.614 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-09-22T02:51:13.630 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-508.el9.x86_64 2024-09-22T02:51:13.630 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T02:51:13.630 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T02:51:13.630 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T02:51:14.631 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-22T02:51:14.631 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T02:51:14.714 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2024-09-22T02:51:14.736 DEBUG:teuthology.parallel:result is None 2024-09-22T02:51:14.736 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-22T02:51:14.748 INFO:teuthology.task.internal:Creating test directory... 2024-09-22T02:51:14.748 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T02:51:14.750 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T02:51:14.770 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-22T02:51:14.776 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-22T02:51:14.783 INFO:teuthology.task.internal:Creating archive directory... 2024-09-22T02:51:14.783 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T02:51:14.809 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T02:51:14.857 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-22T02:51:14.863 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-22T02:51:14.864 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T02:51:14.866 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T02:51:14.896 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T02:51:14.906 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T02:51:14.917 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T02:51:14.929 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T02:51:14.930 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-22T02:51:14.937 INFO:teuthology.task.internal:Configuring sudo... 2024-09-22T02:51:14.937 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T02:51:14.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T02:51:14.999 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-22T02:51:15.009 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-22T02:51:15.009 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T02:51:15.020 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T02:51:15.056 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T02:51:15.118 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T02:51:15.169 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T02:51:15.254 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T02:51:15.294 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:51:15.294 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T02:51:15.360 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T02:51:15.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T02:51:15.455 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T02:51:15.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T02:51:15.583 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:51:15.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T02:51:15.649 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-09-22T02:51:15.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-09-22T02:51:15.699 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T02:51:15.743 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T02:51:16.202 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-22T02:51:16.210 INFO:teuthology.task.internal:Starting timer... 2024-09-22T02:51:16.210 INFO:teuthology.run_tasks:Running task pcp... 2024-09-22T02:51:16.229 INFO:teuthology.run_tasks:Running task selinux... 2024-09-22T02:51:16.237 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-22T02:51:16.237 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2024-09-22T02:51:16.296 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2024-09-22T02:51:16.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-09-22T02:51:16.347 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-09-22T02:51:16.349 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-22T02:51:16.349 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2024-09-22T02:51:16.392 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2024-09-22T02:51:16.393 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-09-22T02:51:16.413 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-09-22T02:51:16.413 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi081.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-09-22T02:51:16.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T02:51:16.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T02:51:16.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T02:51:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T02:51:16.489 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-22T02:51:16.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2024-09-22T02:51:16.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-09-22T02:51:16.571 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-22T02:51:16.579 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-22T02:51:16.582 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-22T02:51:16.761 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-22T02:51:16.771 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-22T02:51:16.772 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-22T02:54:56.567 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-09-22T02:54:56.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-22T02:54:56.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:54:56.643 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-09-22T02:54:56.725 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-09-22T02:54:56.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-22T02:54:56.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:54:56.801 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-22T02:54:56.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-22T02:54:56.877 INFO:teuthology.run_tasks:Running task clock... 2024-09-22T02:54:56.886 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-22T02:54:56.886 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T02:54:56.887 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T02:54:56.889 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T02:54:56.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T02:54:56.915 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T02:54:56.929 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T02:54:56.946 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T02:54:56.953 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2024-09-22T02:54:56.960 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T02:54:56.964 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2024-09-22T02:54:56.978 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T02:54:56.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-09-22T02:54:56.992 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T02:54:56.995 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-09-22T02:54:57.010 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T02:54:57.024 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T02:54:57.047 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-09-22T02:54:57.050 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T02:54:57.050 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-09-22T02:54:57.050 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:54:57.051 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:54:57.051 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:54:57.051 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:54:57.084 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-09-22T02:54:57.088 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T02:54:57.088 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-09-22T02:54:57.088 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:54:57.088 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:54:57.089 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:54:57.089 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:54:57.089 INFO:teuthology.run_tasks:Running task install... 2024-09-22T02:54:57.097 DEBUG:teuthology.task.install:project ceph 2024-09-22T02:54:57.097 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-22T02:54:57.097 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'} 2024-09-22T02:54:57.097 INFO:teuthology.task.install:Using flavor: crimson 2024-09-22T02:54:57.102 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-22T02:54:57.102 INFO:teuthology.task.install:extra packages: [] 2024-09-22T02:54:57.103 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'tag': None, 'wait_for_package': False} 2024-09-22T02:54:57.103 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T02:54:57.104 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'tag': None, 'wait_for_package': False} 2024-09-22T02:54:57.104 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T02:54:57.248 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/ 2024-09-22T02:54:57.248 INFO:teuthology.task.install.rpm:Package version is 19.1.1-373.g4e45e6b4 2024-09-22T02:54:57.251 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/ 2024-09-22T02:54:57.251 INFO:teuthology.task.install.rpm:Package version is 19.1.1-373.g4e45e6b4 2024-09-22T02:54:57.361 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T02:54:57.361 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:54:57.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T02:54:57.370 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T02:54:57.370 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:54:57.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T02:54:57.388 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T02:54:57.388 DEBUG:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e45e6b490e6cbe082cdf155fea780438ec53fc7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T02:54:57.397 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T02:54:57.397 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e45e6b490e6cbe082cdf155fea780438ec53fc7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T02:54:57.454 DEBUG:teuthology.orchestra.run.smithi081:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T02:54:57.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T02:54:57.529 DEBUG:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T02:54:57.536 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T02:54:57.593 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2024-09-22T02:54:57.595 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum clean all 2024-09-22T02:54:57.600 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-09-22T02:54:57.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-09-22T02:54:57.820 INFO:teuthology.orchestra.run.smithi081.stdout:45 files removed 2024-09-22T02:54:57.828 INFO:teuthology.orchestra.run.smithi190.stdout:45 files removed 2024-09-22T02:54:57.846 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw 2024-09-22T02:54:57.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw 2024-09-22T02:54:58.390 INFO:teuthology.orchestra.run.smithi081.stdout:ceph packages for x86_64 332 kB/s | 87 kB 00:00 2024-09-22T02:54:58.409 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 323 kB/s | 87 kB 00:00 2024-09-22T02:54:58.674 INFO:teuthology.orchestra.run.smithi081.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-22T02:54:58.693 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-22T02:54:58.949 INFO:teuthology.orchestra.run.smithi081.stdout:ceph source packages 8.3 kB/s | 2.0 kB 00:00 2024-09-22T02:54:58.969 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 8.3 kB/s | 2.0 kB 00:00 2024-09-22T02:55:00.649 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.3 MB 00:01 2024-09-22T02:55:00.677 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.3 MB 00:01 2024-09-22T02:55:03.967 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 20 MB 00:02 2024-09-22T02:55:08.335 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-22T02:55:09.312 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 20 MB 00:07 2024-09-22T02:55:09.802 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 80 kB/s | 19 kB 00:00 2024-09-22T02:55:10.773 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-22T02:55:13.920 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.5 MB 00:00 2024-09-22T02:55:15.738 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 19 kB 00:00 2024-09-22T02:55:16.162 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-09-22T02:55:16.817 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-22T02:55:18.397 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:55:18.409 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 9.9 M 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.2 M 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.4 M 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 19 M 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 25 k 2024-09-22T02:55:18.410 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 760 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 501 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.1 M 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 45 k 2024-09-22T02:55:18.411 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 136 k 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 161 k 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 321 k 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 301 k 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 99 k 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout:Installing weak dependencies: 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout:Install 30 Packages 2024-09-22T02:55:18.412 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 2 Packages 2024-09-22T02:55:18.413 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:55:18.413 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 58 M 2024-09-22T02:55:18.413 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:55:19.255 INFO:teuthology.orchestra.run.smithi081.stdout:(1/32): ceph-base-19.1.1-373.g4e45e6b4.el9.x86_ 7.1 MB/s | 5.4 MB 00:00 2024-09-22T02:55:19.405 INFO:teuthology.orchestra.run.smithi081.stdout:(2/32): ceph-radosgw-19.1.1-373.g4e45e6b4.el9.x 11 MB/s | 9.9 MB 00:00 2024-09-22T02:55:19.430 INFO:teuthology.orchestra.run.smithi081.stdout:(3/32): ceph-selinux-19.1.1-373.g4e45e6b4.el9.x 143 kB/s | 25 kB 00:00 2024-09-22T02:55:19.464 INFO:teuthology.orchestra.run.smithi081.stdout:(4/32): libcephfs2-19.1.1-373.g4e45e6b4.el9.x86 13 MB/s | 760 kB 00:00 2024-09-22T02:55:19.490 INFO:teuthology.orchestra.run.smithi081.stdout:(5/32): libradosstriper1-19.1.1-373.g4e45e6b4.e 8.4 MB/s | 501 kB 00:00 2024-09-22T02:55:19.540 INFO:teuthology.orchestra.run.smithi081.stdout:(6/32): python3-ceph-argparse-19.1.1-373.g4e45e 907 kB/s | 45 kB 00:00 2024-09-22T02:55:19.723 INFO:teuthology.orchestra.run.smithi081.stdout:(7/32): ceph-common-19.1.1-373.g4e45e6b4.el9.x8 16 MB/s | 19 MB 00:01 2024-09-22T02:55:19.749 INFO:teuthology.orchestra.run.smithi081.stdout:(8/32): python3-ceph-common-19.1.1-373.g4e45e6b 653 kB/s | 136 kB 00:00 2024-09-22T02:55:19.774 INFO:teuthology.orchestra.run.smithi081.stdout:(9/32): python3-cephfs-19.1.1-373.g4e45e6b4.el9 3.1 MB/s | 161 kB 00:00 2024-09-22T02:55:19.799 INFO:teuthology.orchestra.run.smithi081.stdout:(10/32): python3-rados-19.1.1-373.g4e45e6b4.el9 6.2 MB/s | 321 kB 00:00 2024-09-22T02:55:19.833 INFO:teuthology.orchestra.run.smithi081.stdout:(11/32): python3-rbd-19.1.1-373.g4e45e6b4.el9.x 5.0 MB/s | 301 kB 00:00 2024-09-22T02:55:19.858 INFO:teuthology.orchestra.run.smithi081.stdout:(12/32): python3-rgw-19.1.1-373.g4e45e6b4.el9.x 1.7 MB/s | 99 kB 00:00 2024-09-22T02:55:19.933 INFO:teuthology.orchestra.run.smithi081.stdout:(13/32): librgw2-19.1.1-373.g4e45e6b4.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-09-22T02:55:20.309 INFO:teuthology.orchestra.run.smithi081.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 121 kB/s | 45 kB 00:00 2024-09-22T02:55:20.334 INFO:teuthology.orchestra.run.smithi081.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 213 kB/s | 107 kB 00:00 2024-09-22T02:55:20.401 INFO:teuthology.orchestra.run.smithi081.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 295 kB/s | 160 kB 00:00 2024-09-22T02:55:20.518 INFO:teuthology.orchestra.run.smithi081.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 362 kB/s | 42 kB 00:00 2024-09-22T02:55:20.543 INFO:teuthology.orchestra.run.smithi081.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-09-22T02:55:20.660 INFO:teuthology.orchestra.run.smithi081.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 308 kB/s | 36 kB 00:00 2024-09-22T02:55:20.719 INFO:teuthology.orchestra.run.smithi081.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 221 kB 00:00 2024-09-22T02:55:20.769 INFO:teuthology.orchestra.run.smithi081.stdout:(21/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-22T02:55:20.794 INFO:teuthology.orchestra.run.smithi081.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 354 kB/s | 47 kB 00:00 2024-09-22T02:55:20.828 INFO:teuthology.orchestra.run.smithi081.stdout:(23/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.2 MB/s | 662 kB 00:00 2024-09-22T02:55:20.870 INFO:teuthology.orchestra.run.smithi081.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-09-22T02:55:20.903 INFO:teuthology.orchestra.run.smithi081.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 716 kB/s | 49 kB 00:00 2024-09-22T02:55:20.929 INFO:teuthology.orchestra.run.smithi081.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2024-09-22T02:55:20.973 INFO:teuthology.orchestra.run.smithi081.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-22T02:55:21.052 INFO:teuthology.orchestra.run.smithi081.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 839 kB 00:00 2024-09-22T02:55:21.148 INFO:teuthology.orchestra.run.smithi081.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-22T02:55:21.189 INFO:teuthology.orchestra.run.smithi081.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-09-22T02:55:21.380 INFO:teuthology.orchestra.run.smithi081.stdout:(31/32): librados2-19.1.1-373.g4e45e6b4.el9.x86 9.8 MB/s | 3.2 MB 00:00 2024-09-22T02:55:21.431 INFO:teuthology.orchestra.run.smithi081.stdout:(32/32): librbd1-19.1.1-373.g4e45e6b4.el9.x86_6 11 MB/s | 3.1 MB 00:00 2024-09-22T02:55:21.436 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:55:21.436 INFO:teuthology.orchestra.run.smithi081.stdout:Total 19 MB/s | 58 MB 00:03 2024-09-22T02:55:21.539 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:55:21.576 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:55:21.576 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:55:22.245 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:55:22.246 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:55:22.248 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-22T02:55:23.708 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:55:24.086 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-22T02:55:24.223 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T02:55:24.317 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T02:55:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T02:55:24.434 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T02:55:24.474 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:55:24.487 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:55:24.487 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-22T02:55:24.487 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:55:24.487 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:55:24.487 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 9.9 M 2024-09-22T02:55:24.487 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.2 M 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.4 M 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 19 M 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 25 k 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 760 k 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T02:55:24.488 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 501 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.1 M 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 45 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 136 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 161 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 321 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 301 k 2024-09-22T02:55:24.489 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 99 k 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout:Install 30 Packages 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2024-09-22T02:55:24.490 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:55:24.491 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 58 M 2024-09-22T02:55:24.491 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:55:24.510 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T02:55:24.546 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-09-22T02:55:24.580 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-09-22T02:55:24.603 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T02:55:24.626 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T02:55:24.780 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T02:55:24.808 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T02:55:24.837 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-09-22T02:55:24.881 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 10/34 2024-09-22T02:55:24.910 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T02:55:24.985 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 12/34 2024-09-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T02:55:25.104 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T02:55:25.139 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-09-22T02:55:25.233 INFO:teuthology.orchestra.run.smithi190.stdout:(1/32): ceph-base-19.1.1-373.g4e45e6b4.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-09-22T02:55:25.252 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-22T02:55:25.282 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-22T02:55:25.291 INFO:teuthology.orchestra.run.smithi190.stdout:(2/32): ceph-selinux-19.1.1-373.g4e45e6b4.el9.x 433 kB/s | 25 kB 00:00 2024-09-22T02:55:25.324 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-22T02:55:25.356 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-22T02:55:25.374 INFO:teuthology.orchestra.run.smithi190.stdout:(3/32): libcephfs2-19.1.1-373.g4e45e6b4.el9.x86 8.9 MB/s | 760 kB 00:00 2024-09-22T02:55:25.385 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-22T02:55:25.418 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-22T02:55:25.550 INFO:teuthology.orchestra.run.smithi190.stdout:(4/32): ceph-radosgw-19.1.1-373.g4e45e6b4.el9.x 12 MB/s | 9.9 MB 00:00 2024-09-22T02:55:25.575 INFO:teuthology.orchestra.run.smithi190.stdout:(5/32): libradosstriper1-19.1.1-373.g4e45e6b4.e 2.4 MB/s | 501 kB 00:00 2024-09-22T02:55:25.625 INFO:teuthology.orchestra.run.smithi190.stdout:(6/32): python3-ceph-argparse-19.1.1-373.g4e45e 905 kB/s | 45 kB 00:00 2024-09-22T02:55:25.709 INFO:teuthology.orchestra.run.smithi190.stdout:(7/32): python3-ceph-common-19.1.1-373.g4e45e6b 1.6 MB/s | 136 kB 00:00 2024-09-22T02:55:25.752 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-22T02:55:25.789 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T02:55:25.863 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T02:55:25.893 INFO:teuthology.orchestra.run.smithi190.stdout:(8/32): ceph-common-19.1.1-373.g4e45e6b4.el9.x8 17 MB/s | 19 MB 00:01 2024-09-22T02:55:25.894 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/34 2024-09-22T02:55:25.918 INFO:teuthology.orchestra.run.smithi190.stdout:(9/32): python3-cephfs-19.1.1-373.g4e45e6b4.el9 775 kB/s | 161 kB 00:00 2024-09-22T02:55:25.929 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-22T02:55:25.943 INFO:teuthology.orchestra.run.smithi190.stdout:(10/32): python3-rados-19.1.1-373.g4e45e6b4.el9 6.3 MB/s | 321 kB 00:00 2024-09-22T02:55:26.018 INFO:teuthology.orchestra.run.smithi190.stdout:(11/32): librgw2-19.1.1-373.g4e45e6b4.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-09-22T02:55:26.035 INFO:teuthology.orchestra.run.smithi190.stdout:(12/32): python3-rgw-19.1.1-373.g4e45e6b4.el9.x 1.1 MB/s | 99 kB 00:00 2024-09-22T02:55:26.060 INFO:teuthology.orchestra.run.smithi190.stdout:(13/32): python3-rbd-19.1.1-373.g4e45e6b4.el9.x 2.1 MB/s | 301 kB 00:00 2024-09-22T02:55:26.119 INFO:teuthology.orchestra.run.smithi190.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 107 kB 00:00 2024-09-22T02:55:26.144 INFO:teuthology.orchestra.run.smithi190.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 543 kB/s | 45 kB 00:00 2024-09-22T02:55:26.161 INFO:teuthology.orchestra.run.smithi190.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-22T02:55:26.203 INFO:teuthology.orchestra.run.smithi190.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-22T02:55:26.228 INFO:teuthology.orchestra.run.smithi190.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-09-22T02:55:26.254 INFO:teuthology.orchestra.run.smithi190.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-09-22T02:55:26.279 INFO:teuthology.orchestra.run.smithi190.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 709 kB/s | 36 kB 00:00 2024-09-22T02:55:26.296 INFO:teuthology.orchestra.run.smithi190.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-09-22T02:55:26.321 INFO:teuthology.orchestra.run.smithi190.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-22T02:55:26.372 INFO:teuthology.orchestra.run.smithi190.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.3 MB/s | 308 kB 00:00 2024-09-22T02:55:26.389 INFO:teuthology.orchestra.run.smithi190.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2024-09-22T02:55:26.405 INFO:teuthology.orchestra.run.smithi190.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 371 kB/s | 26 kB 00:00 2024-09-22T02:55:26.429 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-22T02:55:26.431 INFO:teuthology.orchestra.run.smithi190.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-22T02:55:26.464 INFO:teuthology.orchestra.run.smithi190.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-22T02:55:26.467 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T02:55:26.506 INFO:teuthology.orchestra.run.smithi190.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-22T02:55:26.541 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T02:55:26.590 INFO:teuthology.orchestra.run.smithi190.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-22T02:55:26.616 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/34 2024-09-22T02:55:26.644 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 28/34 2024-09-22T02:55:26.682 INFO:teuthology.orchestra.run.smithi190.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-22T02:55:26.741 INFO:teuthology.orchestra.run.smithi190.stdout:(31/32): librados2-19.1.1-373.g4e45e6b4.el9.x86 14 MB/s | 3.2 MB 00:00 2024-09-22T02:55:26.799 INFO:teuthology.orchestra.run.smithi190.stdout:(32/32): librbd1-19.1.1-373.g4e45e6b4.el9.x86_6 15 MB/s | 3.1 MB 00:00 2024-09-22T02:55:26.803 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:55:26.803 INFO:teuthology.orchestra.run.smithi190.stdout:Total 25 MB/s | 58 MB 00:02 2024-09-22T02:55:26.902 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:55:26.937 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:55:26.937 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:55:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:55:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:55:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T02:55:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T02:55:29.084 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:55:29.485 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-22T02:55:29.543 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T02:55:29.562 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T02:55:29.602 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T02:55:29.634 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T02:55:29.696 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T02:55:29.696 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T02:55:29.696 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T02:55:29.696 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:55:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T02:55:29.736 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T02:55:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T02:55:29.812 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T02:55:29.840 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-09-22T02:55:29.874 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-09-22T02:55:29.896 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T02:55:29.931 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T02:55:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T02:55:30.139 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T02:55:30.173 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-09-22T02:55:30.216 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 10/34 2024-09-22T02:55:30.245 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T02:55:30.312 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 12/34 2024-09-22T02:55:30.334 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T02:55:30.406 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T02:55:30.442 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-09-22T02:55:30.550 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-22T02:55:30.593 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-22T02:55:30.635 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-22T02:55:30.675 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-22T02:55:30.704 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-22T02:55:30.747 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-22T02:55:31.072 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-22T02:55:31.116 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T02:55:31.207 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T02:55:31.238 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/34 2024-09-22T02:55:31.273 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-22T02:55:31.761 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-22T02:55:31.785 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T02:55:31.840 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T02:55:31.916 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/34 2024-09-22T02:55:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 28/34 2024-09-22T02:55:34.201 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T02:55:34.242 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T02:55:34.822 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T02:55:34.863 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T02:55:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T02:55:34.955 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T02:55:34.955 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T02:55:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:55:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T02:55:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T02:55:39.735 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /sys 2024-09-22T02:55:39.735 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /proc 2024-09-22T02:55:39.735 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /mnt 2024-09-22T02:55:39.735 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /var/tmp 2024-09-22T02:55:39.735 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /home 2024-09-22T02:55:39.735 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /root 2024-09-22T02:55:39.735 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /tmp 2024-09-22T02:55:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:55:40.731 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T02:55:40.756 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T02:55:40.756 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:55:40.756 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T02:55:40.756 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T02:55:40.756 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T02:55:40.756 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:55:40.758 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T02:55:40.841 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T02:55:40.841 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T02:55:42.042 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T02:55:42.043 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/34 2024-09-22T02:55:42.043 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T02:55:42.043 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T02:55:42.043 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T02:55:42.043 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/34 2024-09-22T02:55:42.043 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/34 2024-09-22T02:55:42.043 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-22T02:55:42.043 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 8/34 2024-09-22T02:55:42.043 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 9/34 2024-09-22T02:55:42.043 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 10/34 2024-09-22T02:55:42.043 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-22T02:55:42.044 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-22T02:55:42.045 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-22T02:55:42.045 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-22T02:55:42.045 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-22T02:55:42.045 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-22T02:55:42.045 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-22T02:55:42.045 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T02:55:42.045 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-22T02:55:42.045 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/34 2024-09-22T02:55:43.169 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T02:55:43.170 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:55:43.170 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2024-09-22T02:55:43.170 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.170 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.170 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:55:43.170 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T02:55:43.170 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.170 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.170 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.170 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.170 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T02:55:43.171 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:55:43.172 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:55:43.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-test 2024-09-22T02:55:44.115 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:28 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:55:44.257 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:55:44.258 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:55:44.258 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-22T02:55:44.259 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:55:44.259 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:55:44.259 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 124 M 2024-09-22T02:55:44.259 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-22T02:55:44.259 INFO:teuthology.orchestra.run.smithi081.stdout: libasan x86_64 11.5.0-2.el9 appstream 414 k 2024-09-22T02:55:44.259 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 158 k 2024-09-22T02:55:44.259 INFO:teuthology.orchestra.run.smithi081.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T02:55:44.259 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T02:55:44.259 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T02:55:44.259 INFO:teuthology.orchestra.run.smithi081.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T02:55:44.259 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:55:44.260 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:55:44.260 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:55:44.260 INFO:teuthology.orchestra.run.smithi081.stdout:Install 7 Packages 2024-09-22T02:55:44.260 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:55:44.260 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 125 M 2024-09-22T02:55:44.260 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.1 G 2024-09-22T02:55:44.261 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:55:44.588 INFO:teuthology.orchestra.run.smithi081.stdout:(1/7): libasan-11.5.0-2.el9.x86_64.rpm 1.7 MB/s | 414 kB 00:00 2024-09-22T02:55:44.613 INFO:teuthology.orchestra.run.smithi081.stdout:(2/7): libcephsqlite-19.1.1-373.g4e45e6b4.el9.x 594 kB/s | 158 kB 00:00 2024-09-22T02:55:44.646 INFO:teuthology.orchestra.run.smithi081.stdout:(3/7): libubsan-11.5.0-2.el9.x86_64.rpm 3.1 MB/s | 183 kB 00:00 2024-09-22T02:55:44.697 INFO:teuthology.orchestra.run.smithi081.stdout:(4/7): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-22T02:55:44.747 INFO:teuthology.orchestra.run.smithi081.stdout:(5/7): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.4 MB/s | 122 kB 00:00 2024-09-22T02:55:44.814 INFO:teuthology.orchestra.run.smithi081.stdout:(6/7): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-22T02:55:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T02:55:45.092 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-09-22T02:55:45.092 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-09-22T02:55:45.092 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-09-22T02:55:45.092 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-09-22T02:55:45.092 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-09-22T02:55:45.092 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-09-22T02:55:45.092 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-09-22T02:55:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:55:46.077 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T02:55:46.107 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T02:55:46.107 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:55:46.107 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T02:55:46.107 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T02:55:46.107 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T02:55:46.107 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:55:46.109 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T02:55:46.194 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T02:55:46.195 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T02:55:47.452 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 8/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 9/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 10/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-22T02:55:47.453 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-22T02:55:47.454 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-22T02:55:47.455 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-22T02:55:47.455 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-22T02:55:47.455 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-22T02:55:47.455 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-22T02:55:47.455 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T02:55:47.455 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-22T02:55:47.455 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/34 2024-09-22T02:55:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T02:55:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:55:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T02:55:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T02:55:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T02:55:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T02:55:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T02:55:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:55:48.465 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:55:48.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test 2024-09-22T02:55:49.403 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:27 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:55:49.422 INFO:teuthology.orchestra.run.smithi081.stdout:(7/7): ceph-test-19.1.1-373.g4e45e6b4.el9.x86_6 24 MB/s | 124 MB 00:05 2024-09-22T02:55:49.425 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:55:49.425 INFO:teuthology.orchestra.run.smithi081.stdout:Total 24 MB/s | 125 MB 00:05 2024-09-22T02:55:49.478 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:55:49.499 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:55:49.499 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:55:49.539 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:55:49.540 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:55:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-22T02:55:49.540 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:55:49.540 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:55:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 124 M 2024-09-22T02:55:49.540 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-22T02:55:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: libasan x86_64 11.5.0-2.el9 appstream 414 k 2024-09-22T02:55:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 158 k 2024-09-22T02:55:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T02:55:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T02:55:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T02:55:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T02:55:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:55:49.541 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:55:49.541 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:55:49.541 INFO:teuthology.orchestra.run.smithi190.stdout:Install 7 Packages 2024-09-22T02:55:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:55:49.542 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 125 M 2024-09-22T02:55:49.542 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 1.1 G 2024-09-22T02:55:49.542 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:55:50.084 INFO:teuthology.orchestra.run.smithi190.stdout:(1/7): libcephsqlite-19.1.1-373.g4e45e6b4.el9.x 681 kB/s | 158 kB 00:00 2024-09-22T02:55:50.142 INFO:teuthology.orchestra.run.smithi190.stdout:(2/7): libasan-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 414 kB 00:00 2024-09-22T02:55:50.226 INFO:teuthology.orchestra.run.smithi190.stdout:(3/7): socat-1.7.4.1-6.el9.x86_64.rpm 3.6 MB/s | 304 kB 00:00 2024-09-22T02:55:50.276 INFO:teuthology.orchestra.run.smithi190.stdout:(4/7): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-22T02:55:50.310 INFO:teuthology.orchestra.run.smithi190.stdout:(5/7): libubsan-11.5.0-2.el9.x86_64.rpm 809 kB/s | 183 kB 00:00 2024-09-22T02:55:50.322 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:55:50.323 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:55:50.335 INFO:teuthology.orchestra.run.smithi190.stdout:(6/7): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.0 MB/s | 122 kB 00:00 2024-09-22T02:55:51.385 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:55:51.446 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/7 2024-09-22T02:55:51.490 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/7 2024-09-22T02:55:51.523 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/7 2024-09-22T02:55:51.557 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T02:55:51.597 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libasan-11.5.0-2.el9.x86_64 5/7 2024-09-22T02:55:51.625 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T02:55:53.277 INFO:teuthology.orchestra.run.smithi190.stdout:(7/7): ceph-test-19.1.1-373.g4e45e6b4.el9.x86_6 36 MB/s | 124 MB 00:03 2024-09-22T02:55:53.279 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:55:53.280 INFO:teuthology.orchestra.run.smithi190.stdout:Total 34 MB/s | 125 MB 00:03 2024-09-22T02:55:53.334 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:55:53.354 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:55:53.355 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:55:54.174 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:55:54.175 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:55:55.232 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:55:55.317 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/7 2024-09-22T02:55:55.361 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/7 2024-09-22T02:55:55.397 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/7 2024-09-22T02:55:55.428 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T02:55:55.469 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libasan-11.5.0-2.el9.x86_64 5/7 2024-09-22T02:55:55.497 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T02:56:08.785 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T02:56:08.857 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T02:56:12.760 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T02:56:12.836 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T02:56:17.423 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T02:56:17.423 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/7 2024-09-22T02:56:17.424 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/7 2024-09-22T02:56:17.424 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libasan-11.5.0-2.el9.x86_64 3/7 2024-09-22T02:56:17.424 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T02:56:17.424 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 5/7 2024-09-22T02:56:17.424 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/7 2024-09-22T02:56:17.809 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 7/7 2024-09-22T02:56:17.809 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:56:17.809 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:56:17.809 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:56:17.809 INFO:teuthology.orchestra.run.smithi081.stdout: libasan-11.5.0-2.el9.x86_64 2024-09-22T02:56:17.809 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:56:17.809 INFO:teuthology.orchestra.run.smithi081.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T02:56:17.809 INFO:teuthology.orchestra.run.smithi081.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T02:56:17.810 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T02:56:17.810 INFO:teuthology.orchestra.run.smithi081.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T02:56:17.810 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:56:17.810 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:56:18.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph 2024-09-22T02:56:18.696 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:02 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:56:18.830 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:56:18.833 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:56:18.833 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-22T02:56:18.833 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:56:18.833 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:56:18.833 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 6.6 k 2024-09-22T02:56:18.833 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-22T02:56:18.833 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 2.3 M 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 1.5 M 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 249 k 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 4.9 M 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 120 M 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T02:56:18.834 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T02:56:18.835 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:56:18.836 INFO:teuthology.orchestra.run.smithi081.stdout:Install 38 Packages 2024-09-22T02:56:18.837 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:56:18.837 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 134 M 2024-09-22T02:56:18.837 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.4 G 2024-09-22T02:56:18.837 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:56:19.483 INFO:teuthology.orchestra.run.smithi081.stdout:(1/38): ceph-19.1.1-373.g4e45e6b4.el9.x86_64.rp 47 kB/s | 6.6 kB 00:00 2024-09-22T02:56:19.650 INFO:teuthology.orchestra.run.smithi081.stdout:(2/38): ceph-mgr-19.1.1-373.g4e45e6b4.el9.x86_6 4.9 MB/s | 1.5 MB 00:00 2024-09-22T02:56:19.700 INFO:teuthology.orchestra.run.smithi081.stdout:(3/38): ceph-mds-19.1.1-373.g4e45e6b4.el9.x86_6 6.4 MB/s | 2.3 MB 00:00 2024-09-22T02:56:19.792 INFO:teuthology.orchestra.run.smithi081.stdout:(4/38): ceph-mgr-modules-core-19.1.1-373.g4e45e 2.7 MB/s | 249 kB 00:00 2024-09-22T02:56:19.859 INFO:teuthology.orchestra.run.smithi081.stdout:(5/38): ceph-mon-19.1.1-373.g4e45e6b4.el9.x86_6 13 MB/s | 4.9 MB 00:00 2024-09-22T02:56:19.968 INFO:teuthology.orchestra.run.smithi081.stdout:(6/38): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-22T02:56:20.051 INFO:teuthology.orchestra.run.smithi081.stdout:(7/38): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 106 kB 00:00 2024-09-22T02:56:20.110 INFO:teuthology.orchestra.run.smithi081.stdout:(8/38): python3-cryptography-36.0.1-4.el9.x86_6 4.9 MB/s | 1.2 MB 00:00 2024-09-22T02:56:20.135 INFO:teuthology.orchestra.run.smithi081.stdout:(9/38): python3-pycparser-2.20-6.el9.noarch.rpm 1.6 MB/s | 135 kB 00:00 2024-09-22T02:56:20.185 INFO:teuthology.orchestra.run.smithi081.stdout:(10/38): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-09-22T02:56:20.219 INFO:teuthology.orchestra.run.smithi081.stdout:(11/38): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-22T02:56:20.952 INFO:teuthology.orchestra.run.smithi081.stdout:(12/38): lua-5.4.4-4.el9.x86_64.rpm 246 kB/s | 188 kB 00:00 2024-09-22T02:56:21.127 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T02:56:21.127 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/7 2024-09-22T02:56:21.127 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/7 2024-09-22T02:56:21.127 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libasan-11.5.0-2.el9.x86_64 3/7 2024-09-22T02:56:21.127 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T02:56:21.127 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 5/7 2024-09-22T02:56:21.127 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/7 2024-09-22T02:56:21.136 INFO:teuthology.orchestra.run.smithi081.stdout:(13/38): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-09-22T02:56:21.499 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 7/7 2024-09-22T02:56:21.500 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:56:21.500 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:56:21.500 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:56:21.500 INFO:teuthology.orchestra.run.smithi190.stdout: libasan-11.5.0-2.el9.x86_64 2024-09-22T02:56:21.500 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:56:21.500 INFO:teuthology.orchestra.run.smithi190.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T02:56:21.500 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T02:56:21.500 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T02:56:21.500 INFO:teuthology.orchestra.run.smithi190.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T02:56:21.500 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:56:21.500 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:56:21.686 INFO:teuthology.orchestra.run.smithi081.stdout:(14/38): python3-markupsafe-1.1.1-12.el9.x86_64 63 kB/s | 35 kB 00:00 2024-09-22T02:56:21.711 INFO:teuthology.orchestra.run.smithi081.stdout:(15/38): python3-mako-1.1.4-6.el9.noarch.rpm 227 kB/s | 172 kB 00:00 2024-09-22T02:56:21.814 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph 2024-09-22T02:56:22.391 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:00 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:56:22.533 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:56:22.698 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:56:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-22T02:56:22.698 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:56:22.698 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:56:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 6.6 k 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 2.3 M 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 1.5 M 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 249 k 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 4.9 M 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 120 M 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T02:56:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T02:56:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:56:22.701 INFO:teuthology.orchestra.run.smithi190.stdout:Install 38 Packages 2024-09-22T02:56:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:56:22.702 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 134 M 2024-09-22T02:56:22.702 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 1.4 G 2024-09-22T02:56:22.702 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:56:22.863 INFO:teuthology.orchestra.run.smithi190.stdout:(1/38): ceph-19.1.1-373.g4e45e6b4.el9.x86_64.rp 51 kB/s | 6.6 kB 00:00 2024-09-22T02:56:22.887 INFO:teuthology.orchestra.run.smithi081.stdout:(16/38): ceph-osd-19.1.1-373.g4e45e6b4.el9.x86_ 37 MB/s | 120 MB 00:03 2024-09-22T02:56:23.071 INFO:teuthology.orchestra.run.smithi190.stdout:(2/38): ceph-mgr-19.1.1-373.g4e45e6b4.el9.x86_6 4.5 MB/s | 1.5 MB 00:00 2024-09-22T02:56:23.305 INFO:teuthology.orchestra.run.smithi190.stdout:(3/38): ceph-mds-19.1.1-373.g4e45e6b4.el9.x86_6 4.0 MB/s | 2.3 MB 00:00 2024-09-22T02:56:23.345 INFO:teuthology.orchestra.run.smithi081.stdout:(17/38): python3-packaging-20.9-5.el9.noarch.rp 47 kB/s | 77 kB 00:01 2024-09-22T02:56:23.372 INFO:teuthology.orchestra.run.smithi190.stdout:(4/38): ceph-mgr-modules-core-19.1.1-373.g4e45e 3.7 MB/s | 249 kB 00:00 2024-09-22T02:56:23.404 INFO:teuthology.orchestra.run.smithi081.stdout:(18/38): python3-pytz-2021.1-5.el9.noarch.rpm 30 kB/s | 51 kB 00:01 2024-09-22T02:56:23.454 INFO:teuthology.orchestra.run.smithi081.stdout:(19/38): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-22T02:56:23.479 INFO:teuthology.orchestra.run.smithi081.stdout:(20/38): python3-toml-0.10.2-6.el9.noarch.rpm 71 kB/s | 42 kB 00:00 2024-09-22T02:56:23.513 INFO:teuthology.orchestra.run.smithi081.stdout:(21/38): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 754 kB/s | 43 kB 00:00 2024-09-22T02:56:23.539 INFO:teuthology.orchestra.run.smithi190.stdout:(5/38): ceph-mon-19.1.1-373.g4e45e6b4.el9.x86_6 7.2 MB/s | 4.9 MB 00:00 2024-09-22T02:56:23.563 INFO:teuthology.orchestra.run.smithi081.stdout:(22/38): python3-cherrypy-18.6.1-2.el9.noarch.r 8.3 MB/s | 358 kB 00:00 2024-09-22T02:56:23.573 INFO:teuthology.orchestra.run.smithi190.stdout:(6/38): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-22T02:56:23.589 INFO:teuthology.orchestra.run.smithi081.stdout:(23/38): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-09-22T02:56:23.614 INFO:teuthology.orchestra.run.smithi081.stdout:(24/38): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-22T02:56:23.631 INFO:teuthology.orchestra.run.smithi190.stdout:(7/38): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-22T02:56:23.647 INFO:teuthology.orchestra.run.smithi081.stdout:(25/38): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-09-22T02:56:23.676 INFO:teuthology.orchestra.run.smithi081.stdout:(26/38): python3-jaraco-collections-3.0.0-8.el9 427 kB/s | 23 kB 00:00 2024-09-22T02:56:23.690 INFO:teuthology.orchestra.run.smithi190.stdout:(8/38): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-09-22T02:56:23.701 INFO:teuthology.orchestra.run.smithi081.stdout:(27/38): python3-jaraco-functools-3.5.0-2.el9.n 363 kB/s | 19 kB 00:00 2024-09-22T02:56:23.727 INFO:teuthology.orchestra.run.smithi081.stdout:(28/38): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-22T02:56:23.744 INFO:teuthology.orchestra.run.smithi081.stdout:(29/38): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-09-22T02:56:23.744 INFO:teuthology.orchestra.run.smithi190.stdout:(9/38): python3-requests-2.25.1-8.el9.noarch.rp 2.3 MB/s | 125 kB 00:00 2024-09-22T02:56:23.769 INFO:teuthology.orchestra.run.smithi081.stdout:(30/38): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-22T02:56:23.786 INFO:teuthology.orchestra.run.smithi081.stdout:(31/38): python3-portend-3.1.0-2.el9.noarch.rpm 1.0 MB/s | 16 kB 00:00 2024-09-22T02:56:23.794 INFO:teuthology.orchestra.run.smithi190.stdout:(10/38): python3-cryptography-36.0.1-4.el9.x86_ 4.8 MB/s | 1.2 MB 00:00 2024-09-22T02:56:23.819 INFO:teuthology.orchestra.run.smithi190.stdout:(11/38): python3-urllib3-1.26.5-6.el9.noarch.rp 2.8 MB/s | 215 kB 00:00 2024-09-22T02:56:23.819 INFO:teuthology.orchestra.run.smithi081.stdout:(32/38): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.7 MB/s | 90 kB 00:00 2024-09-22T02:56:23.845 INFO:teuthology.orchestra.run.smithi081.stdout:(33/38): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-09-22T02:56:23.862 INFO:teuthology.orchestra.run.smithi081.stdout:(34/38): python3-tempora-5.0.0-2.el9.noarch.rpm 855 kB/s | 36 kB 00:00 2024-09-22T02:56:23.895 INFO:teuthology.orchestra.run.smithi081.stdout:(35/38): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-22T02:56:23.921 INFO:teuthology.orchestra.run.smithi081.stdout:(36/38): python3-zc-lockfile-2.0-10.el9.noarch. 791 kB/s | 20 kB 00:00 2024-09-22T02:56:23.954 INFO:teuthology.orchestra.run.smithi081.stdout:(37/38): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-22T02:56:23.979 INFO:teuthology.orchestra.run.smithi081.stdout:(38/38): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-09-22T02:56:23.985 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:56:23.986 INFO:teuthology.orchestra.run.smithi081.stdout:Total 26 MB/s | 134 MB 00:05 2024-09-22T02:56:24.028 INFO:teuthology.orchestra.run.smithi190.stdout:(12/38): lua-5.4.4-4.el9.x86_64.rpm 807 kB/s | 188 kB 00:00 2024-09-22T02:56:24.111 INFO:teuthology.orchestra.run.smithi190.stdout:(13/38): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-22T02:56:24.151 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:56:24.153 INFO:teuthology.orchestra.run.smithi190.stdout:(14/38): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-09-22T02:56:24.178 INFO:teuthology.orchestra.run.smithi190.stdout:(15/38): python3-markupsafe-1.1.1-12.el9.x86_64 520 kB/s | 35 kB 00:00 2024-09-22T02:56:24.192 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:56:24.192 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:56:24.212 INFO:teuthology.orchestra.run.smithi190.stdout:(16/38): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-09-22T02:56:24.237 INFO:teuthology.orchestra.run.smithi190.stdout:(17/38): python3-pytz-2021.1-5.el9.noarch.rpm 869 kB/s | 51 kB 00:00 2024-09-22T02:56:24.271 INFO:teuthology.orchestra.run.smithi190.stdout:(18/38): python3-toml-0.10.2-6.el9.noarch.rpm 711 kB/s | 42 kB 00:00 2024-09-22T02:56:24.313 INFO:teuthology.orchestra.run.smithi190.stdout:(19/38): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-22T02:56:24.338 INFO:teuthology.orchestra.run.smithi190.stdout:(20/38): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-22T02:56:24.363 INFO:teuthology.orchestra.run.smithi190.stdout:(21/38): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.8 MB/s | 172 kB 00:00 2024-09-22T02:56:24.397 INFO:teuthology.orchestra.run.smithi190.stdout:(22/38): python3-cherrypy-18.6.1-2.el9.noarch.r 10 MB/s | 358 kB 00:00 2024-09-22T02:56:24.422 INFO:teuthology.orchestra.run.smithi190.stdout:(23/38): python3-jaraco-8.2.1-3.el9.noarch.rpm 431 kB/s | 11 kB 00:00 2024-09-22T02:56:24.448 INFO:teuthology.orchestra.run.smithi190.stdout:(24/38): python3-jaraco-classes-3.2.1-5.el9.noa 705 kB/s | 18 kB 00:00 2024-09-22T02:56:24.473 INFO:teuthology.orchestra.run.smithi190.stdout:(25/38): python3-jaraco-collections-3.0.0-8.el9 927 kB/s | 23 kB 00:00 2024-09-22T02:56:24.507 INFO:teuthology.orchestra.run.smithi190.stdout:(26/38): python3-jaraco-functools-3.5.0-2.el9.n 749 kB/s | 19 kB 00:00 2024-09-22T02:56:24.532 INFO:teuthology.orchestra.run.smithi190.stdout:(27/38): python3-jaraco-text-3.2.0-6.el9.noarch 790 kB/s | 20 kB 00:00 2024-09-22T02:56:24.557 INFO:teuthology.orchestra.run.smithi190.stdout:(28/38): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-22T02:56:24.582 INFO:teuthology.orchestra.run.smithi190.stdout:(29/38): python3-more-itertools-8.12.0-2.el9.no 3.2 MB/s | 79 kB 00:00 2024-09-22T02:56:24.616 INFO:teuthology.orchestra.run.smithi190.stdout:(30/38): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-09-22T02:56:24.650 INFO:teuthology.orchestra.run.smithi190.stdout:(31/38): python3-portend-3.1.0-2.el9.noarch.rpm 490 kB/s | 16 kB 00:00 2024-09-22T02:56:24.684 INFO:teuthology.orchestra.run.smithi190.stdout:(32/38): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2024-09-22T02:56:24.709 INFO:teuthology.orchestra.run.smithi190.stdout:(33/38): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-22T02:56:24.734 INFO:teuthology.orchestra.run.smithi190.stdout:(34/38): python3-webob-1.8.8-2.el9.noarch.rpm 9.0 MB/s | 230 kB 00:00 2024-09-22T02:56:24.768 INFO:teuthology.orchestra.run.smithi190.stdout:(35/38): python3-werkzeug-2.0.3-3.el9.1.noarch. 13 MB/s | 427 kB 00:00 2024-09-22T02:56:24.818 INFO:teuthology.orchestra.run.smithi190.stdout:(36/38): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-09-22T02:56:24.843 INFO:teuthology.orchestra.run.smithi190.stdout:(37/38): lua-devel-5.4.4-4.el9.x86_64.rpm 37 kB/s | 22 kB 00:00 2024-09-22T02:56:25.256 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:56:25.257 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:56:27.077 INFO:teuthology.orchestra.run.smithi190.stdout:(38/38): ceph-osd-19.1.1-373.g4e45e6b4.el9.x86_ 30 MB/s | 120 MB 00:04 2024-09-22T02:56:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:56:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:Total 30 MB/s | 134 MB 00:04 2024-09-22T02:56:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:56:27.263 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:56:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:56:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:56:27.324 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/38 2024-09-22T02:56:27.366 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/38 2024-09-22T02:56:27.393 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/38 2024-09-22T02:56:27.465 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/38 2024-09-22T02:56:27.517 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/38 2024-09-22T02:56:27.549 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/38 2024-09-22T02:56:27.642 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/38 2024-09-22T02:56:27.729 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/38 2024-09-22T02:56:27.768 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/38 2024-09-22T02:56:27.804 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/38 2024-09-22T02:56:27.834 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/38 2024-09-22T02:56:27.868 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/38 2024-09-22T02:56:27.902 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/38 2024-09-22T02:56:27.942 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/38 2024-09-22T02:56:27.977 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/38 2024-09-22T02:56:28.011 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/38 2024-09-22T02:56:28.060 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/38 2024-09-22T02:56:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/38 2024-09-22T02:56:28.173 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/38 2024-09-22T02:56:28.320 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/38 2024-09-22T02:56:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/38 2024-09-22T02:56:28.372 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:56:28.373 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:56:30.352 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:56:30.421 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/38 2024-09-22T02:56:30.456 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/38 2024-09-22T02:56:30.490 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/38 2024-09-22T02:56:30.563 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/38 2024-09-22T02:56:30.606 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/38 2024-09-22T02:56:30.639 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/38 2024-09-22T02:56:30.731 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/38 2024-09-22T02:56:30.819 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/38 2024-09-22T02:56:30.866 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/38 2024-09-22T02:56:30.902 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/38 2024-09-22T02:56:30.931 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/38 2024-09-22T02:56:30.966 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/38 2024-09-22T02:56:30.999 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/38 2024-09-22T02:56:31.040 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/38 2024-09-22T02:56:31.083 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/38 2024-09-22T02:56:31.117 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/38 2024-09-22T02:56:31.160 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/38 2024-09-22T02:56:31.192 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/38 2024-09-22T02:56:31.265 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/38 2024-09-22T02:56:31.405 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/38 2024-09-22T02:56:31.459 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/38 2024-09-22T02:56:47.296 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/38 2024-09-22T02:56:47.329 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T02:56:47.358 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T02:56:47.358 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:56:47.359 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T02:56:47.359 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T02:56:47.359 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T02:56:47.359 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:56:47.462 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 24/38 2024-09-22T02:56:47.510 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/38 2024-09-22T02:56:47.567 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ply-3.11-14.el9.noarch 26/38 2024-09-22T02:56:47.622 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 27/38 2024-09-22T02:56:47.811 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 28/38 2024-09-22T02:56:47.864 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 29/38 2024-09-22T02:56:47.942 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/38 2024-09-22T02:56:48.033 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 31/38 2024-09-22T02:56:48.073 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 32/38 2024-09-22T02:56:48.192 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 33/38 2024-09-22T02:56:48.358 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 34/38 2024-09-22T02:56:48.384 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T02:56:48.407 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T02:56:48.407 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:56:48.407 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T02:56:48.407 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T02:56:48.407 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T02:56:48.407 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:56:48.885 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T02:56:48.908 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T02:56:48.908 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:56:48.908 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T02:56:48.908 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T02:56:48.908 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T02:56:48.908 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:56:49.135 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T02:56:49.158 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T02:56:49.158 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:56:49.158 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T02:56:49.158 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T02:56:49.158 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T02:56:49.158 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:56:49.225 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T02:56:50.285 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/38 2024-09-22T02:56:50.311 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T02:56:50.340 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T02:56:50.340 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:56:50.340 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T02:56:50.340 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T02:56:50.340 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T02:56:50.341 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:56:50.444 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 24/38 2024-09-22T02:56:50.483 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/38 2024-09-22T02:56:50.541 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 26/38 2024-09-22T02:56:50.596 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 27/38 2024-09-22T02:56:50.782 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 28/38 2024-09-22T02:56:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 29/38 2024-09-22T02:56:50.908 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/38 2024-09-22T02:56:51.005 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 31/38 2024-09-22T02:56:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 32/38 2024-09-22T02:56:51.162 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 33/38 2024-09-22T02:56:51.327 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 34/38 2024-09-22T02:56:51.355 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T02:56:51.379 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T02:56:51.379 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:56:51.379 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T02:56:51.379 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T02:56:51.379 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T02:56:51.379 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:56:51.839 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T02:56:51.863 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T02:56:51.863 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:56:51.863 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T02:56:51.863 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T02:56:51.863 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T02:56:51.863 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:56:52.115 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T02:56:52.138 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T02:56:52.138 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:56:52.138 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T02:56:52.139 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T02:56:52.139 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T02:56:52.139 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:56:52.203 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T02:56:59.664 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T02:56:59.664 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/38 2024-09-22T02:56:59.664 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/38 2024-09-22T02:56:59.664 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/38 2024-09-22T02:56:59.664 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/38 2024-09-22T02:56:59.664 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/38 2024-09-22T02:56:59.664 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 6/38 2024-09-22T02:56:59.664 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/38 2024-09-22T02:56:59.664 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/38 2024-09-22T02:56:59.664 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 14/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 15/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 16/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 17/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 18/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 19/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 20/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 21/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 22/38 2024-09-22T02:56:59.665 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 23/38 2024-09-22T02:56:59.666 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 24/38 2024-09-22T02:56:59.666 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 25/38 2024-09-22T02:56:59.666 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 26/38 2024-09-22T02:56:59.666 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 27/38 2024-09-22T02:56:59.666 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 28/38 2024-09-22T02:56:59.667 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 29/38 2024-09-22T02:56:59.667 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 30/38 2024-09-22T02:56:59.667 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 31/38 2024-09-22T02:56:59.667 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 32/38 2024-09-22T02:56:59.667 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 33/38 2024-09-22T02:56:59.667 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 34/38 2024-09-22T02:56:59.667 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 35/38 2024-09-22T02:56:59.668 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 36/38 2024-09-22T02:56:59.668 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 37/38 2024-09-22T02:57:00.591 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 38/38 2024-09-22T02:57:00.591 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:00.591 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:00.591 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:00.591 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:00.591 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T02:57:00.592 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T02:57:00.593 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T02:57:00.594 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T02:57:00.594 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T02:57:00.594 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T02:57:00.594 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T02:57:00.594 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T02:57:00.594 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T02:57:00.594 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T02:57:00.594 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T02:57:00.594 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:00.594 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:00.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-base 2024-09-22T02:57:01.497 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:45 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:01.589 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:01.633 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:01.634 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-22T02:57:01.634 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:01.702 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install cephadm 2024-09-22T02:57:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T02:57:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/38 2024-09-22T02:57:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/38 2024-09-22T02:57:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/38 2024-09-22T02:57:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/38 2024-09-22T02:57:02.077 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 6/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 14/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 15/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 16/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 17/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 18/38 2024-09-22T02:57:02.078 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 19/38 2024-09-22T02:57:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 20/38 2024-09-22T02:57:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 21/38 2024-09-22T02:57:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 22/38 2024-09-22T02:57:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 23/38 2024-09-22T02:57:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 24/38 2024-09-22T02:57:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 25/38 2024-09-22T02:57:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 26/38 2024-09-22T02:57:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 27/38 2024-09-22T02:57:02.079 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 28/38 2024-09-22T02:57:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 29/38 2024-09-22T02:57:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 30/38 2024-09-22T02:57:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 31/38 2024-09-22T02:57:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 32/38 2024-09-22T02:57:02.080 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 33/38 2024-09-22T02:57:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 34/38 2024-09-22T02:57:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 35/38 2024-09-22T02:57:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 36/38 2024-09-22T02:57:02.081 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 37/38 2024-09-22T02:57:02.280 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:46 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:02.412 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:02.412 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:02.412 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-22T02:57:02.413 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:02.413 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:02.413 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 765 k 2024-09-22T02:57:02.413 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:02.413 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:02.413 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:02.413 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-22T02:57:02.413 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:02.413 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 765 k 2024-09-22T02:57:02.413 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 771 k 2024-09-22T02:57:02.414 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:02.743 INFO:teuthology.orchestra.run.smithi081.stdout:cephadm-19.1.1-373.g4e45e6b4.el9.noarch.rpm 2.3 MB/s | 765 kB 00:00 2024-09-22T02:57:02.746 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:02.746 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.3 MB/s | 765 kB 00:00 2024-09-22T02:57:02.746 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:02.749 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:02.749 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:02.755 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:02.755 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:02.825 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:02.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 38/38 2024-09-22T02:57:02.986 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:02.986 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T02:57:02.987 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T02:57:02.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:02.989 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:03.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base 2024-09-22T02:57:03.328 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:03.421 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:03.767 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:03.840 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:41 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:03.930 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:03.971 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:03.972 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-22T02:57:03.973 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:04.007 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:04.007 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:04.007 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:04.007 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:04.008 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:04.008 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:04.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install cephadm 2024-09-22T02:57:04.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T02:57:04.611 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:42 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:04.741 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:04.742 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:04.743 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-22T02:57:04.743 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:04.743 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:04.743 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 765 k 2024-09-22T02:57:04.743 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:04.743 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:04.744 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:04.744 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-22T02:57:04.744 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:04.744 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 765 k 2024-09-22T02:57:04.744 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 771 k 2024-09-22T02:57:04.744 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:04.813 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:48 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:04.944 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:04.946 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:04.946 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-09-22T02:57:04.947 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:04.947 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:04.947 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 147 k 2024-09-22T02:57:04.947 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:04.947 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:04.948 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:04.948 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-22T02:57:04.948 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:04.948 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 147 k 2024-09-22T02:57:04.949 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 481 k 2024-09-22T02:57:04.949 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:05.041 INFO:teuthology.orchestra.run.smithi190.stdout:cephadm-19.1.1-373.g4e45e6b4.el9.noarch.rpm 2.5 MB/s | 765 kB 00:00 2024-09-22T02:57:05.042 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:05.042 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.5 MB/s | 765 kB 00:00 2024-09-22T02:57:05.042 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:05.047 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:05.047 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:05.052 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:05.053 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:05.130 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:05.158 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-immutable-object-cache-19.1.1-373.g4e45e6b 696 kB/s | 147 kB 00:00 2024-09-22T02:57:05.158 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:05.159 INFO:teuthology.orchestra.run.smithi081.stdout:Total 691 kB/s | 147 kB 00:00 2024-09-22T02:57:05.159 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:05.167 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:05.167 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:05.224 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:05.225 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:05.377 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T02:57:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T02:57:05.484 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:57:05.484 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T02:57:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:05.601 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:05.703 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:06.070 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:06.314 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:06.314 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:06.315 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:06.388 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T02:57:06.388 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:06.388 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:06.388 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:06.388 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:06.388 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:06.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T02:57:06.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr 2024-09-22T02:57:07.148 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:45 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:07.262 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:51 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:07.284 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:07.285 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:07.285 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-22T02:57:07.285 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:07.285 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:07.285 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 147 k 2024-09-22T02:57:07.285 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:07.285 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:07.286 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:07.286 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-22T02:57:07.286 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:07.286 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 147 k 2024-09-22T02:57:07.286 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 481 k 2024-09-22T02:57:07.286 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:07.352 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:07.394 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:07.395 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-22T02:57:07.396 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:07.465 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T02:57:07.497 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-19.1.1-373.g4e45e6b 698 kB/s | 147 kB 00:00 2024-09-22T02:57:07.497 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:07.498 INFO:teuthology.orchestra.run.smithi190.stdout:Total 693 kB/s | 147 kB 00:00 2024-09-22T02:57:07.498 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:07.505 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:07.505 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:07.562 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:07.562 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:07.714 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:07.773 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T02:57:07.798 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T02:57:07.798 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:57:07.798 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T02:57:07.799 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:08.039 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:52 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:08.170 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:08.171 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:08.171 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-22T02:57:08.171 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:08.171 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:08.171 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 3.6 M 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 32 k 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 16 k 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:08.172 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:08.173 INFO:teuthology.orchestra.run.smithi081.stdout:Install 11 Packages 2024-09-22T02:57:08.173 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:08.173 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 7.7 M 2024-09-22T02:57:08.173 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 101 M 2024-09-22T02:57:08.173 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:08.720 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T02:57:08.720 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:08.721 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:08.721 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:08.721 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:08.721 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:08.828 INFO:teuthology.orchestra.run.smithi081.stdout:(1/11): ceph-grafana-dashboards-19.1.1-373.g4e4 210 kB/s | 32 kB 00:00 2024-09-22T02:57:08.862 INFO:teuthology.orchestra.run.smithi081.stdout:(2/11): ceph-prometheus-alerts-19.1.1-373.g4e45 89 kB/s | 16 kB 00:00 2024-09-22T02:57:08.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr 2024-09-22T02:57:09.071 INFO:teuthology.orchestra.run.smithi081.stdout:(3/11): ceph-mgr-dashboard-19.1.1-373.g4e45e6b4 9.1 MB/s | 3.6 MB 00:00 2024-09-22T02:57:09.104 INFO:teuthology.orchestra.run.smithi081.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.5 MB/s | 863 kB 00:00 2024-09-22T02:57:09.155 INFO:teuthology.orchestra.run.smithi081.stdout:(5/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 6.4 MB/s | 548 kB 00:00 2024-09-22T02:57:09.180 INFO:teuthology.orchestra.run.smithi081.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 287 kB/s | 19 kB 00:00 2024-09-22T02:57:09.214 INFO:teuthology.orchestra.run.smithi081.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-22T02:57:09.239 INFO:teuthology.orchestra.run.smithi081.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-22T02:57:09.273 INFO:teuthology.orchestra.run.smithi081.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-09-22T02:57:09.314 INFO:teuthology.orchestra.run.smithi081.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 13 MB/s | 2.0 MB 00:00 2024-09-22T02:57:09.414 INFO:teuthology.orchestra.run.smithi081.stdout:(11/11): python3-protobuf-3.14.0-13.el9.noarch. 460 kB/s | 269 kB 00:00 2024-09-22T02:57:09.418 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:09.418 INFO:teuthology.orchestra.run.smithi081.stdout:Total 6.2 MB/s | 7.7 MB 00:01 2024-09-22T02:57:09.448 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:09.467 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:09.467 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:09.536 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:47 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:09.625 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:09.668 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:09.669 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-22T02:57:09.669 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:09.672 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:09.672 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:09.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T02:57:10.162 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:10.253 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T02:57:10.284 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T02:57:10.306 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:48 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:10.396 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T02:57:10.438 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:10.439 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:10.439 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-22T02:57:10.439 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:10.439 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:10.439 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 3.6 M 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 32 k 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 16 k 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:10.440 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:10.441 INFO:teuthology.orchestra.run.smithi190.stdout:Install 11 Packages 2024-09-22T02:57:10.441 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:10.441 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 7.7 M 2024-09-22T02:57:10.441 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 101 M 2024-09-22T02:57:10.441 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:10.518 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T02:57:10.571 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T02:57:10.663 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T02:57:10.723 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T02:57:10.752 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T02:57:10.783 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 9/11 2024-09-22T02:57:10.992 INFO:teuthology.orchestra.run.smithi190.stdout:(1/11): ceph-grafana-dashboards-19.1.1-373.g4e4 238 kB/s | 32 kB 00:00 2024-09-22T02:57:11.009 INFO:teuthology.orchestra.run.smithi190.stdout:(2/11): ceph-prometheus-alerts-19.1.1-373.g4e45 109 kB/s | 16 kB 00:00 2024-09-22T02:57:11.135 INFO:teuthology.orchestra.run.smithi190.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-22T02:57:11.219 INFO:teuthology.orchestra.run.smithi190.stdout:(4/11): ceph-mgr-dashboard-19.1.1-373.g4e45e6b4 9.9 MB/s | 3.6 MB 00:00 2024-09-22T02:57:11.252 INFO:teuthology.orchestra.run.smithi190.stdout:(5/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.6 MB/s | 548 kB 00:00 2024-09-22T02:57:11.277 INFO:teuthology.orchestra.run.smithi190.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-09-22T02:57:11.311 INFO:teuthology.orchestra.run.smithi190.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-22T02:57:11.336 INFO:teuthology.orchestra.run.smithi190.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-22T02:57:11.362 INFO:teuthology.orchestra.run.smithi190.stdout:(9/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-09-22T02:57:11.395 INFO:teuthology.orchestra.run.smithi190.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-09-22T02:57:11.437 INFO:teuthology.orchestra.run.smithi190.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-22T02:57:11.440 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:11.441 INFO:teuthology.orchestra.run.smithi190.stdout:Total 7.7 MB/s | 7.7 MB 00:01 2024-09-22T02:57:11.471 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:11.488 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:11.488 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:11.687 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:11.687 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:11.861 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 10/11 2024-09-22T02:57:11.901 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T02:57:12.184 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:12.259 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T02:57:12.290 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T02:57:12.407 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T02:57:12.540 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T02:57:12.593 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T02:57:12.686 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T02:57:12.729 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T02:57:12.766 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T02:57:12.798 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 9/11 2024-09-22T02:57:13.184 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T02:57:13.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 1/11 2024-09-22T02:57:13.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 2/11 2024-09-22T02:57:13.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 3/11 2024-09-22T02:57:13.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T02:57:13.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T02:57:13.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T02:57:13.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T02:57:13.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T02:57:13.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T02:57:13.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T02:57:13.494 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T02:57:13.494 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:13.494 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:13.494 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T02:57:13.494 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:13.494 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:13.494 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:13.494 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T02:57:13.494 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T02:57:13.495 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T02:57:13.495 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T02:57:13.495 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T02:57:13.495 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T02:57:13.495 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T02:57:13.495 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:13.495 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:13.749 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T02:57:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 10/11 2024-09-22T02:57:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T02:57:14.469 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:58 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:14.601 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:14.602 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-09-22T02:57:14.602 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-22T02:57:14.602 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 7.4 M 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T02:57:14.603 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T02:57:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T02:57:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:14.604 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:14.604 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-09-22T02:57:14.604 INFO:teuthology.orchestra.run.smithi081.stdout:Install 12 Packages 2024-09-22T02:57:14.604 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:14.604 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 38 M 2024-09-22T02:57:14.605 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 212 M 2024-09-22T02:57:14.605 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:15.089 INFO:teuthology.orchestra.run.smithi081.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 910 kB/s | 191 kB 00:00 2024-09-22T02:57:15.189 INFO:teuthology.orchestra.run.smithi081.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.5 MB/s | 801 kB 00:00 2024-09-22T02:57:15.210 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T02:57:15.210 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 1/11 2024-09-22T02:57:15.210 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 2/11 2024-09-22T02:57:15.210 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 3/11 2024-09-22T02:57:15.210 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T02:57:15.210 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T02:57:15.210 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T02:57:15.210 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T02:57:15.210 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T02:57:15.210 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T02:57:15.211 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T02:57:15.214 INFO:teuthology.orchestra.run.smithi081.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-09-22T02:57:15.264 INFO:teuthology.orchestra.run.smithi081.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-22T02:57:15.373 INFO:teuthology.orchestra.run.smithi081.stdout:(5/12): ceph-mgr-diskprediction-local-19.1.1-37 15 MB/s | 7.4 MB 00:00 2024-09-22T02:57:15.398 INFO:teuthology.orchestra.run.smithi081.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 288 kB/s | 38 kB 00:00 2024-09-22T02:57:15.498 INFO:teuthology.orchestra.run.smithi081.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-09-22T02:57:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T02:57:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:15.527 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T02:57:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T02:57:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T02:57:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T02:57:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T02:57:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T02:57:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T02:57:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T02:57:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:15.528 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:15.682 INFO:teuthology.orchestra.run.smithi081.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2024-09-22T02:57:15.749 INFO:teuthology.orchestra.run.smithi081.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-09-22T02:57:15.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T02:57:15.774 INFO:teuthology.orchestra.run.smithi081.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.7 MB/s | 250 kB 00:00 2024-09-22T02:57:15.833 INFO:teuthology.orchestra.run.smithi081.stdout:(11/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 4.6 MB/s | 3.0 MB 00:00 2024-09-22T02:57:16.083 INFO:teuthology.orchestra.run.smithi081.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-09-22T02:57:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:Total 26 MB/s | 38 MB 00:01 2024-09-22T02:57:16.342 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:54 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:16.435 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:16.460 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:16.461 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:16.474 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:16.475 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-09-22T02:57:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-22T02:57:16.475 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-09-22T02:57:16.475 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 7.4 M 2024-09-22T02:57:16.475 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-22T02:57:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T02:57:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout:Install 12 Packages 2024-09-22T02:57:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:16.477 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 38 M 2024-09-22T02:57:16.477 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 212 M 2024-09-22T02:57:16.477 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:16.894 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:16.894 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:16.928 INFO:teuthology.orchestra.run.smithi190.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 854 kB/s | 191 kB 00:00 2024-09-22T02:57:17.028 INFO:teuthology.orchestra.run.smithi190.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.4 MB/s | 801 kB 00:00 2024-09-22T02:57:17.054 INFO:teuthology.orchestra.run.smithi190.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-09-22T02:57:17.129 INFO:teuthology.orchestra.run.smithi190.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 243 kB/s | 18 kB 00:00 2024-09-22T02:57:17.221 INFO:teuthology.orchestra.run.smithi190.stdout:(5/12): ceph-mgr-diskprediction-local-19.1.1-37 14 MB/s | 7.4 MB 00:00 2024-09-22T02:57:17.246 INFO:teuthology.orchestra.run.smithi190.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 329 kB/s | 38 kB 00:00 2024-09-22T02:57:17.355 INFO:teuthology.orchestra.run.smithi190.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-09-22T02:57:17.651 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:17.723 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T02:57:17.763 INFO:teuthology.orchestra.run.smithi190.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-09-22T02:57:17.772 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T02:57:17.847 INFO:teuthology.orchestra.run.smithi190.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.0 MB/s | 250 kB 00:00 2024-09-22T02:57:17.882 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T02:57:17.916 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T02:57:18.099 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T02:57:18.131 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T02:57:18.208 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T02:57:18.330 INFO:teuthology.orchestra.run.smithi190.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.2 MB/s | 5.1 MB 00:00 2024-09-22T02:57:18.397 INFO:teuthology.orchestra.run.smithi190.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.2 MB/s | 4.9 MB 00:01 2024-09-22T02:57:18.863 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T02:57:18.979 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T02:57:20.156 INFO:teuthology.orchestra.run.smithi190.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.0 MB/s | 16 MB 00:02 2024-09-22T02:57:20.157 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:20.157 INFO:teuthology.orchestra.run.smithi190.stdout:Total 10 MB/s | 38 MB 00:03 2024-09-22T02:57:20.222 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T02:57:20.514 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:20.541 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:20.541 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:20.965 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:20.965 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:21.244 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T02:57:21.273 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T02:57:21.730 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T02:57:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T02:57:21.949 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T02:57:21.997 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T02:57:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T02:57:22.229 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T02:57:22.305 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T02:57:22.953 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T02:57:23.059 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T02:57:23.850 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T02:57:23.851 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 1/12 2024-09-22T02:57:23.851 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T02:57:23.851 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T02:57:23.851 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T02:57:23.851 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T02:57:23.851 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T02:57:23.851 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T02:57:23.851 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T02:57:23.851 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T02:57:23.851 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T02:57:23.851 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T02:57:24.285 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T02:57:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T02:57:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:24.304 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T02:57:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T02:57:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T02:57:24.304 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T02:57:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T02:57:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T02:57:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T02:57:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T02:57:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T02:57:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T02:57:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T02:57:24.305 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:24.305 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:24.711 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-rook 2024-09-22T02:57:25.294 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:09 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:25.312 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T02:57:25.346 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T02:57:25.427 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:25.428 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:25.428 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-22T02:57:25.428 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:25.428 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:25.428 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 49 k 2024-09-22T02:57:25.428 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-22T02:57:25.428 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T02:57:25.428 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:25.429 INFO:teuthology.orchestra.run.smithi081.stdout:Install 13 Packages 2024-09-22T02:57:25.430 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:25.430 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 2.2 M 2024-09-22T02:57:25.430 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 26 M 2024-09-22T02:57:25.430 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:25.699 INFO:teuthology.orchestra.run.smithi081.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 337 kB/s | 26 kB 00:00 2024-09-22T02:57:25.724 INFO:teuthology.orchestra.run.smithi081.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 189 kB/s | 19 kB 00:00 2024-09-22T02:57:25.791 INFO:teuthology.orchestra.run.smithi081.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-09-22T02:57:25.816 INFO:teuthology.orchestra.run.smithi081.stdout:(4/13): ceph-mgr-rook-19.1.1-373.g4e45e6b4.el9. 254 kB/s | 49 kB 00:00 2024-09-22T02:57:25.841 INFO:teuthology.orchestra.run.smithi081.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-22T02:57:25.883 INFO:teuthology.orchestra.run.smithi081.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 772 kB/s | 32 kB 00:00 2024-09-22T02:57:25.908 INFO:teuthology.orchestra.run.smithi081.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 582 kB/s | 54 kB 00:00 2024-09-22T02:57:25.934 INFO:teuthology.orchestra.run.smithi081.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 280 kB/s | 14 kB 00:00 2024-09-22T02:57:25.967 INFO:teuthology.orchestra.run.smithi081.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-22T02:57:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.1 MB/s | 220 kB 00:00 2024-09-22T02:57:26.034 INFO:teuthology.orchestra.run.smithi081.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-22T02:57:26.059 INFO:teuthology.orchestra.run.smithi081.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 654 kB/s | 59 kB 00:00 2024-09-22T02:57:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-09-22T02:57:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-22T02:57:26.152 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:26.167 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:26.167 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:26.254 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:26.256 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:26.759 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:26.915 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T02:57:26.970 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T02:57:27.024 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T02:57:27.071 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T02:57:27.112 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T02:57:27.222 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T02:57:27.332 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T02:57:27.389 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T02:57:27.836 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T02:57:27.836 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 1/12 2024-09-22T02:57:27.837 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T02:57:27.837 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T02:57:27.837 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T02:57:27.837 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T02:57:27.837 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T02:57:27.837 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T02:57:27.837 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T02:57:27.837 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T02:57:27.837 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T02:57:27.838 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T02:57:28.092 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T02:57:28.137 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T02:57:28.170 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T02:57:28.218 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T02:57:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T02:57:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T02:57:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T02:57:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T02:57:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:28.229 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:28.240 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T02:57:28.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-rook 2024-09-22T02:57:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T02:57:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 1/13 2024-09-22T02:57:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T02:57:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T02:57:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T02:57:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T02:57:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T02:57:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T02:57:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T02:57:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T02:57:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T02:57:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T02:57:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T02:57:29.201 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:07 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:29.335 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:29.336 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 49 k 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T02:57:29.337 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T02:57:29.338 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T02:57:29.338 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T02:57:29.338 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:29.338 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:29.338 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:29.338 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2024-09-22T02:57:29.338 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:29.338 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 2.2 M 2024-09-22T02:57:29.339 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 26 M 2024-09-22T02:57:29.339 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:29.348 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T02:57:29.349 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T02:57:29.350 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T02:57:29.350 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T02:57:29.350 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T02:57:29.350 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:29.350 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:29.583 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T02:57:30.039 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 254 kB/s | 19 kB 00:00 2024-09-22T02:57:30.065 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 255 kB/s | 26 kB 00:00 2024-09-22T02:57:30.131 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-09-22T02:57:30.157 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-22T02:57:30.163 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:14 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:30.190 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.6 MB/s | 279 kB 00:00 2024-09-22T02:57:30.215 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): ceph-mgr-rook-19.1.1-373.g4e45e6b4.el9. 196 kB/s | 49 kB 00:00 2024-09-22T02:57:30.241 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 639 kB/s | 54 kB 00:00 2024-09-22T02:57:30.291 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-22T02:57:30.295 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:30.295 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:30.295 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-22T02:57:30.295 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:30.296 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:30.296 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 146 k 2024-09-22T02:57:30.296 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-22T02:57:30.296 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T02:57:30.296 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T02:57:30.296 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T02:57:30.296 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T02:57:30.296 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T02:57:30.297 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:30.297 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:30.297 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:30.297 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2024-09-22T02:57:30.297 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:30.297 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 7.0 M 2024-09-22T02:57:30.297 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 33 M 2024-09-22T02:57:30.298 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:30.316 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-22T02:57:30.342 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-22T02:57:30.367 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-22T02:57:30.392 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-22T02:57:30.426 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-22T02:57:30.430 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:30.430 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.0 MB/s | 2.2 MB 00:01 2024-09-22T02:57:30.497 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:30.512 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:30.512 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:30.582 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-22T02:57:30.601 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:30.602 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:30.604 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): ceph-mgr-cephadm-19.1.1-373.g4e45e6b4.el 577 kB/s | 146 kB 00:00 2024-09-22T02:57:30.655 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.4 MB/s | 548 kB 00:00 2024-09-22T02:57:30.680 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 765 kB/s | 58 kB 00:00 2024-09-22T02:57:30.705 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-22T02:57:31.106 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:31.264 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T02:57:31.310 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T02:57:31.366 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T02:57:31.403 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T02:57:31.444 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T02:57:31.546 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T02:57:31.648 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T02:57:31.696 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T02:57:32.407 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T02:57:32.447 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.8 MB/s | 6.0 MB 00:02 2024-09-22T02:57:32.451 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:32.451 INFO:teuthology.orchestra.run.smithi081.stdout:Total 3.3 MB/s | 7.0 MB 00:02 2024-09-22T02:57:32.452 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T02:57:32.486 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T02:57:32.525 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:32.533 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T02:57:32.537 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:32.537 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:32.555 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T02:57:32.637 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:32.637 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:32.941 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:33.111 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T02:57:33.157 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T02:57:33.337 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T02:57:33.337 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 1/13 2024-09-22T02:57:33.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T02:57:33.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T02:57:33.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T02:57:33.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T02:57:33.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T02:57:33.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T02:57:33.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T02:57:33.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T02:57:33.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T02:57:33.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T02:57:33.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T02:57:33.658 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T02:57:33.658 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:33.659 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:33.659 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:33.659 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T02:57:33.659 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T02:57:33.659 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T02:57:33.659 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T02:57:33.659 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T02:57:33.659 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T02:57:33.659 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T02:57:33.659 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T02:57:33.659 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T02:57:33.660 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T02:57:33.660 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T02:57:33.660 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T02:57:33.660 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:33.660 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T02:57:33.784 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T02:57:33.857 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T02:57:33.886 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T02:57:33.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T02:57:34.465 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:12 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:34.598 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:34.598 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-22T02:57:34.598 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:34.598 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 146 k 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-09-22T02:57:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:34.600 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 7.0 M 2024-09-22T02:57:34.600 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 33 M 2024-09-22T02:57:34.600 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:34.692 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T02:57:34.692 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/6 2024-09-22T02:57:34.692 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T02:57:34.693 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T02:57:34.693 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T02:57:34.693 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T02:57:34.803 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-09-22T02:57:34.853 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-22T02:57:34.879 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-22T02:57:34.904 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): ceph-mgr-cephadm-19.1.1-373.g4e45e6b4.el 640 kB/s | 146 kB 00:00 2024-09-22T02:57:34.929 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-22T02:57:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T02:57:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:34.967 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T02:57:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T02:57:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T02:57:34.967 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T02:57:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T02:57:34.968 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:34.968 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:35.054 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-22T02:57:35.056 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:35.056 INFO:teuthology.orchestra.run.smithi190.stdout:Total 15 MB/s | 7.0 MB 00:00 2024-09-22T02:57:35.125 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:35.136 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:35.136 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:35.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-fuse 2024-09-22T02:57:35.235 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:35.235 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:35.552 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:35.712 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T02:57:35.764 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T02:57:35.819 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:19 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:35.954 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:35.954 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:35.954 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-22T02:57:35.954 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:35.954 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:35.954 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 868 k 2024-09-22T02:57:35.954 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-09-22T02:57:35.955 INFO:teuthology.orchestra.run.smithi081.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T02:57:35.955 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:35.955 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:35.955 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:35.955 INFO:teuthology.orchestra.run.smithi081.stdout:Install 2 Packages 2024-09-22T02:57:35.955 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:35.955 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 948 k 2024-09-22T02:57:35.955 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 3.1 M 2024-09-22T02:57:35.956 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:36.100 INFO:teuthology.orchestra.run.smithi081.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 649 kB/s | 80 kB 00:00 2024-09-22T02:57:36.267 INFO:teuthology.orchestra.run.smithi081.stdout:(2/2): ceph-fuse-19.1.1-373.g4e45e6b4.el9.x86_6 2.9 MB/s | 868 kB 00:00 2024-09-22T02:57:36.267 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:36.268 INFO:teuthology.orchestra.run.smithi081.stdout:Total 3.0 MB/s | 948 kB 00:00 2024-09-22T02:57:36.279 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:36.280 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T02:57:36.300 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:36.300 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:36.361 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T02:57:36.379 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:36.379 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:36.431 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T02:57:36.460 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T02:57:36.567 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:36.704 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T02:57:36.774 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T02:57:37.231 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T02:57:37.231 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/6 2024-09-22T02:57:37.231 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T02:57:37.231 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T02:57:37.231 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T02:57:37.231 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T02:57:37.502 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T02:57:37.502 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/2 2024-09-22T02:57:37.524 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T02:57:37.525 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:37.525 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:37.525 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:37.525 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T02:57:37.525 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T02:57:37.525 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T02:57:37.525 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T02:57:37.525 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T02:57:37.525 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:37.525 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:37.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse 2024-09-22T02:57:37.772 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T02:57:37.772 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:37.772 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:37.772 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T02:57:37.772 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:37.772 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:38.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-volume 2024-09-22T02:57:38.355 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:16 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:38.490 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:38.491 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:38.491 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-22T02:57:38.491 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:38.491 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:38.491 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 868 k 2024-09-22T02:57:38.491 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-09-22T02:57:38.491 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T02:57:38.491 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:38.491 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:38.491 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:38.492 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-09-22T02:57:38.492 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:38.492 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 948 k 2024-09-22T02:57:38.492 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 3.1 M 2024-09-22T02:57:38.492 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:38.645 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:22 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:38.787 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:38.788 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-09-22T02:57:38.788 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:38.788 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 285 k 2024-09-22T02:57:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:38.788 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:38.788 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:38.788 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-22T02:57:38.788 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:38.789 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 285 k 2024-09-22T02:57:38.789 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.3 M 2024-09-22T02:57:38.789 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:38.899 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 531 kB/s | 80 kB 00:00 2024-09-22T02:57:39.022 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-volume-19.1.1-373.g4e45e6b4.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-22T02:57:39.023 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:39.023 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-22T02:57:39.023 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:39.024 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): ceph-fuse-19.1.1-373.g4e45e6b4.el9.x86_6 3.1 MB/s | 868 kB 00:00 2024-09-22T02:57:39.024 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:39.025 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.7 MB/s | 948 kB 00:00 2024-09-22T02:57:39.031 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:39.032 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:39.037 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:39.056 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:39.056 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:39.059 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:39.059 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:39.140 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:39.140 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:39.275 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:39.349 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:39.365 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:39.365 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:57:39.365 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T02:57:39.365 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T02:57:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T02:57:40.207 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T02:57:40.207 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/2 2024-09-22T02:57:40.407 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:40.407 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:40.407 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:40.408 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:40.408 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:40.408 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:40.492 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T02:57:40.492 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:40.492 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:40.493 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T02:57:40.493 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:40.493 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:40.721 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados-devel 2024-09-22T02:57:40.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-volume 2024-09-22T02:57:41.318 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:25 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:41.363 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:19 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:41.452 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:41.452 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:41.452 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-09-22T02:57:41.452 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:41.452 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:41.452 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 127 k 2024-09-22T02:57:41.452 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:41.452 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:41.452 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:41.453 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-22T02:57:41.453 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:41.453 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 127 k 2024-09-22T02:57:41.453 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 457 k 2024-09-22T02:57:41.453 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:41.500 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:41.501 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-09-22T02:57:41.501 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:41.501 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 285 k 2024-09-22T02:57:41.501 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:41.501 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:41.502 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:41.502 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-22T02:57:41.502 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:41.502 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 285 k 2024-09-22T02:57:41.502 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 1.3 M 2024-09-22T02:57:41.502 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:41.661 INFO:teuthology.orchestra.run.smithi081.stdout:librados-devel-19.1.1-373.g4e45e6b4.el9.x86_64. 610 kB/s | 127 kB 00:00 2024-09-22T02:57:41.662 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:41.662 INFO:teuthology.orchestra.run.smithi081.stdout:Total 606 kB/s | 127 kB 00:00 2024-09-22T02:57:41.662 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:41.668 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:41.668 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:41.728 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-volume-19.1.1-373.g4e45e6b4.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-22T02:57:41.729 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:41.729 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-22T02:57:41.729 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:41.731 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:41.731 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:41.738 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:41.738 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:41.769 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:41.769 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:41.890 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:41.984 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:41.988 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:42.064 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:42.081 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:42.081 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:57:42.081 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T02:57:42.081 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:42.409 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:42.663 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:42.663 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:42.663 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:42.663 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:42.663 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:42.663 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:42.912 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs2 2024-09-22T02:57:43.091 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T02:57:43.091 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:43.091 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:43.091 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T02:57:43.091 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:43.091 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:43.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel 2024-09-22T02:57:43.506 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:27 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:43.598 INFO:teuthology.orchestra.run.smithi081.stdout:Package libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:43.645 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:43.646 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-22T02:57:43.646 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:43.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs-devel 2024-09-22T02:57:43.977 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:21 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:44.110 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:44.110 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:44.110 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-22T02:57:44.110 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:44.111 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:44.111 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 127 k 2024-09-22T02:57:44.111 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:44.111 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:44.111 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:44.111 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-22T02:57:44.111 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:44.112 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 127 k 2024-09-22T02:57:44.112 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 457 k 2024-09-22T02:57:44.112 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:44.288 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:28 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:44.317 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-19.1.1-373.g4e45e6b4.el9.x86_64. 616 kB/s | 127 kB 00:00 2024-09-22T02:57:44.318 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:44.318 INFO:teuthology.orchestra.run.smithi190.stdout:Total 612 kB/s | 127 kB 00:00 2024-09-22T02:57:44.318 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:44.324 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:44.324 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:44.388 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:44.388 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:44.421 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:44.421 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:44.421 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-09-22T02:57:44.421 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:44.422 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:44.422 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 32 k 2024-09-22T02:57:44.422 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:44.422 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:44.422 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:44.422 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-22T02:57:44.422 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:44.423 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 32 k 2024-09-22T02:57:44.423 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 142 k 2024-09-22T02:57:44.423 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:44.537 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:44.582 INFO:teuthology.orchestra.run.smithi081.stdout:libcephfs-devel-19.1.1-373.g4e45e6b4.el9.x86_64 199 kB/s | 32 kB 00:00 2024-09-22T02:57:44.582 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:44.583 INFO:teuthology.orchestra.run.smithi081.stdout:Total 197 kB/s | 32 kB 00:00 2024-09-22T02:57:44.583 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:44.586 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:44.586 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:44.603 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:44.603 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:44.641 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:44.696 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:44.804 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:45.156 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:45.348 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:45.348 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:45.348 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:45.388 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:45.388 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:45.388 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:45.388 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:45.388 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:45.388 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:45.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2 2024-09-22T02:57:45.594 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados2 2024-09-22T02:57:46.164 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:24 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:46.182 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:30 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:46.255 INFO:teuthology.orchestra.run.smithi190.stdout:Package libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:46.272 INFO:teuthology.orchestra.run.smithi081.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:46.299 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:46.301 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-22T02:57:46.301 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:46.315 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:46.316 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-22T02:57:46.317 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:46.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs-devel 2024-09-22T02:57:46.384 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librbd1 2024-09-22T02:57:46.943 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:24 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:46.971 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:30 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:47.062 INFO:teuthology.orchestra.run.smithi081.stdout:Package librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:47.075 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:47.075 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:47.076 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-22T02:57:47.076 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:47.076 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:47.076 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 32 k 2024-09-22T02:57:47.076 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:47.076 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:47.076 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:47.076 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-22T02:57:47.076 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:47.077 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 32 k 2024-09-22T02:57:47.077 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 142 k 2024-09-22T02:57:47.077 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:47.107 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:47.108 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-22T02:57:47.108 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:47.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rados 2024-09-22T02:57:47.230 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs-devel-19.1.1-373.g4e45e6b4.el9.x86_64 207 kB/s | 32 kB 00:00 2024-09-22T02:57:47.231 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:47.231 INFO:teuthology.orchestra.run.smithi190.stdout:Total 205 kB/s | 32 kB 00:00 2024-09-22T02:57:47.231 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:47.235 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:47.235 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:47.257 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:47.257 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:47.350 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:47.452 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:47.756 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:31 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:47.801 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:47.845 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:47.889 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:47.890 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-22T02:57:47.890 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:47.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rgw 2024-09-22T02:57:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:48.038 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:48.039 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:48.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2 2024-09-22T02:57:48.539 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:32 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:48.628 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:48.672 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:48.673 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-22T02:57:48.673 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:48.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-cephfs 2024-09-22T02:57:48.793 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:26 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:48.884 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:48.928 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:48.929 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-22T02:57:48.929 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:48.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1 2024-09-22T02:57:49.320 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:33 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:49.408 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:49.451 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:49.452 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-22T02:57:49.452 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:49.528 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rbd 2024-09-22T02:57:49.579 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:27 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:49.669 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:49.713 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:49.714 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-22T02:57:49.714 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:49.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados 2024-09-22T02:57:50.104 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:34 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:50.192 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:50.236 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:50.237 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-09-22T02:57:50.237 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:50.308 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-fuse 2024-09-22T02:57:50.358 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:28 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:50.448 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:50.492 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:50.493 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-22T02:57:50.493 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:50.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw 2024-09-22T02:57:50.891 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:34 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:51.025 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:51.025 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:51.026 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-09-22T02:57:51.026 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:51.026 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:51.026 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 85 k 2024-09-22T02:57:51.026 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:51.026 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:51.026 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:51.026 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-22T02:57:51.026 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:51.027 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 85 k 2024-09-22T02:57:51.027 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 256 k 2024-09-22T02:57:51.027 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:51.138 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:29 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:51.219 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-fuse-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 446 kB/s | 85 kB 00:00 2024-09-22T02:57:51.219 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:51.219 INFO:teuthology.orchestra.run.smithi081.stdout:Total 443 kB/s | 85 kB 00:00 2024-09-22T02:57:51.219 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:51.226 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:51.226 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:51.228 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:51.272 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:51.273 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-22T02:57:51.273 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:51.291 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:51.291 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:51.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs 2024-09-22T02:57:51.443 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:51.549 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:51.908 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:29 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:51.936 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:51.998 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:52.042 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:52.043 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-22T02:57:52.043 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:52.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd 2024-09-22T02:57:52.175 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:52.175 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:52.176 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:52.176 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:52.176 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:52.176 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:52.444 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-mirror 2024-09-22T02:57:52.691 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:30 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:52.780 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T02:57:52.823 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:52.824 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-09-22T02:57:52.825 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:52.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse 2024-09-22T02:57:53.036 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:37 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:53.174 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 3.1 M 2024-09-22T02:57:53.175 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 14 M 2024-09-22T02:57:53.176 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:53.466 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:31 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:53.506 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-mirror-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 9.2 MB/s | 3.1 MB 00:00 2024-09-22T02:57:53.507 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:53.507 INFO:teuthology.orchestra.run.smithi081.stdout:Total 9.2 MB/s | 3.1 MB 00:00 2024-09-22T02:57:53.507 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:53.516 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:53.516 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:53.596 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:53.596 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:53.600 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:53.601 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:53.601 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-09-22T02:57:53.601 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:53.601 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:53.601 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 85 k 2024-09-22T02:57:53.601 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:53.601 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:53.601 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:53.601 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-22T02:57:53.601 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:53.602 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 85 k 2024-09-22T02:57:53.602 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 256 k 2024-09-22T02:57:53.602 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:53.791 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 452 kB/s | 85 kB 00:00 2024-09-22T02:57:53.792 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:53.792 INFO:teuthology.orchestra.run.smithi190.stdout:Total 449 kB/s | 85 kB 00:00 2024-09-22T02:57:53.792 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:53.799 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:53.799 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:53.863 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:53.864 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:54.013 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:54.046 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:54.107 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:54.122 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:54.132 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:54.132 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:57:54.132 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T02:57:54.132 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T02:57:54.132 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T02:57:54.132 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:54.478 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:54.747 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:54.747 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:54.747 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:54.747 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:54.747 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:54.747 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:55.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2024-09-22T02:57:55.221 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:55.222 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:55.222 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:55.222 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:55.222 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:55.222 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:55.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-nbd 2024-09-22T02:57:55.633 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:33 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:55.776 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.1 M 2024-09-22T02:57:55.777 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 14 M 2024-09-22T02:57:55.778 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:56.044 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:40 ago on Sun 22 Sep 2024 02:55:16 AM UTC. 2024-09-22T02:57:56.105 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 9.3 MB/s | 3.1 MB 00:00 2024-09-22T02:57:56.105 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:56.106 INFO:teuthology.orchestra.run.smithi190.stdout:Total 9.3 MB/s | 3.1 MB 00:00 2024-09-22T02:57:56.106 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:56.115 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:56.115 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:56.181 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-09-22T02:57:56.182 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:56.182 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-09-22T02:57:56.182 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:56.182 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-09-22T02:57:56.182 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 169 k 2024-09-22T02:57:56.182 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:56.182 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-09-22T02:57:56.182 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-09-22T02:57:56.182 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-09-22T02:57:56.182 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:56.183 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 169 k 2024-09-22T02:57:56.183 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 540 k 2024-09-22T02:57:56.183 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-09-22T02:57:56.199 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:56.199 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:56.427 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-nbd-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 695 kB/s | 169 kB 00:00 2024-09-22T02:57:56.427 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:56.428 INFO:teuthology.orchestra.run.smithi081.stdout:Total 692 kB/s | 169 kB 00:00 2024-09-22T02:57:56.428 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-09-22T02:57:56.435 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-09-22T02:57:56.435 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-09-22T02:57:56.499 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-09-22T02:57:56.499 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-09-22T02:57:56.650 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-09-22T02:57:56.676 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:56.747 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:56.757 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:56.773 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:57:56.773 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T02:57:56.773 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T02:57:56.774 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T02:57:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:57.156 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:57.430 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:57.430 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:57.431 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-09-22T02:57:57.431 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:57.431 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:57:57.431 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-09-22T02:57:57.687 DEBUG:teuthology.parallel:result is None 2024-09-22T02:57:57.873 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:57:57.873 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:57.873 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:57:57.873 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:57:57.873 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:57.874 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:57:58.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd 2024-09-22T02:57:58.731 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:36 ago on Sun 22 Sep 2024 02:55:22 AM UTC. 2024-09-22T02:57:58.866 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-09-22T02:57:58.866 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:58.866 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-09-22T02:57:58.866 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:58.866 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-09-22T02:57:58.866 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 169 k 2024-09-22T02:57:58.866 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:58.867 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-09-22T02:57:58.867 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-09-22T02:57:58.867 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-09-22T02:57:58.867 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:57:58.867 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 169 k 2024-09-22T02:57:58.867 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 540 k 2024-09-22T02:57:58.868 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-09-22T02:57:59.292 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 399 kB/s | 169 kB 00:00 2024-09-22T02:57:59.292 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:59.293 INFO:teuthology.orchestra.run.smithi190.stdout:Total 398 kB/s | 169 kB 00:00 2024-09-22T02:57:59.293 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-09-22T02:57:59.300 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-09-22T02:57:59.301 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-09-22T02:57:59.365 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-09-22T02:57:59.366 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-09-22T02:57:59.522 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-09-22T02:57:59.632 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:58:00.032 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:58:00.296 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T02:58:00.296 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:00.296 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-09-22T02:58:00.297 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:58:00.297 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:00.297 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-09-22T02:58:00.563 DEBUG:teuthology.parallel:result is None 2024-09-22T02:58:00.563 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T02:58:00.708 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T02:58:00.734 INFO:teuthology.orchestra.run.smithi081.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-22T02:58:00.734 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373.g4e45e6b4.el9 2024-09-22T02:58:00.734 INFO:teuthology.task.install:The correct ceph version 19.1.1-373.g4e45e6b4 is installed. 2024-09-22T02:58:00.736 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T02:58:00.880 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T02:58:00.902 INFO:teuthology.orchestra.run.smithi190.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-22T02:58:00.903 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373.g4e45e6b4.el9 2024-09-22T02:58:00.903 INFO:teuthology.task.install:The correct ceph version 19.1.1-373.g4e45e6b4 is installed. 2024-09-22T02:58:00.904 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-22T02:58:00.904 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:00.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T02:58:00.935 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:00.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T02:58:00.969 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-22T02:58:00.970 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:00.970 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T02:58:01.005 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T02:58:01.072 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:01.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T02:58:01.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T02:58:01.166 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-22T02:58:01.167 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:01.167 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T02:58:01.195 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T02:58:01.262 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:01.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T02:58:01.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T02:58:01.354 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-22T02:58:01.354 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:01.354 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T02:58:01.381 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T02:58:01.447 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:01.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T02:58:01.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T02:58:01.537 INFO:teuthology.run_tasks:Running task ceph... 2024-09-22T02:58:01.610 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-22T02:58:01.610 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /var/log/ceph 2024-09-22T02:58:01.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2024-09-22T02:58:01.637 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-22T02:58:01.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T02:58:01.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T02:58:01.703 INFO:tasks.ceph:Creating extra log directories... 2024-09-22T02:58:01.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T02:58:01.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T02:58:01.778 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-22T02:58:01.778 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-22T02:58:01.778 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914712', 'branch': 'squid', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914712', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'squid', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 1158, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'targets': {'smithi081.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBC7zrS1ffvustjUUkTcbVOw2gZT7UUiNJzcSUenkt3NmUxvIgPkBgAlZ6j97Cw8EPvxVDHNzjmjy6LDCX3auKA=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2evev0l2v39HqdhVel1zzTPuwFmZzH+X42VuM4ETyQs1pX/AYtv05/u2TpbJogC5QzSHBF8V/wVvU5UL8NWrM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'radosbench': {'clients': ['client.0'], 'time': 150}}, {'admin_socket': {'client.0': {'objecter_requests': {'test': 'http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_21:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-22T02:58:01.779 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T02:58:01.812 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T02:58:01.827 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T02:58:01.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T02:58:01.892 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:01.893 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T02:58:01.947 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T02:58:01.947 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-09-22T02:58:02.003 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T02:58:02.003 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T02:58:02.003 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-22T02:58:02.003 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T02:58:02.003 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T02:58:02.003 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-09-22 02:57:54.380057099 +0000 2024-09-22T02:58:02.003 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-09-22 02:54:35.806030384 +0000 2024-09-22T02:58:02.003 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-09-22 02:54:35.806030384 +0000 2024-09-22T02:58:02.003 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-09-22 02:54:35.806030384 +0000 2024-09-22T02:58:02.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T02:58:02.066 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-09-22T02:58:02.066 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-09-22T02:58:02.066 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000204494 s, 2.5 MB/s 2024-09-22T02:58:02.067 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T02:58:02.124 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-09-22T02:58:02.178 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T02:58:02.179 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T02:58:02.179 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-22T02:58:02.179 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T02:58:02.179 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T02:58:02.179 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-09-22 02:57:54.381057080 +0000 2024-09-22T02:58:02.179 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-09-22 02:54:36.133023950 +0000 2024-09-22T02:58:02.179 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-09-22 02:54:36.133023950 +0000 2024-09-22T02:58:02.179 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-09-22 02:54:36.133023950 +0000 2024-09-22T02:58:02.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T02:58:02.241 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-09-22T02:58:02.241 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-09-22T02:58:02.241 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000195621 s, 2.6 MB/s 2024-09-22T02:58:02.242 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T02:58:02.298 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-09-22T02:58:02.353 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T02:58:02.353 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T02:58:02.354 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-22T02:58:02.354 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T02:58:02.354 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T02:58:02.354 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-09-22 02:57:54.381057080 +0000 2024-09-22T02:58:02.354 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-09-22 02:54:36.461017497 +0000 2024-09-22T02:58:02.354 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-09-22 02:54:36.461017497 +0000 2024-09-22T02:58:02.354 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-09-22 02:54:36.461017497 +0000 2024-09-22T02:58:02.354 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T02:58:02.416 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-09-22T02:58:02.416 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-09-22T02:58:02.416 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000205855 s, 2.5 MB/s 2024-09-22T02:58:02.418 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T02:58:02.474 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-09-22T02:58:02.528 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T02:58:02.528 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T02:58:02.528 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 727 Links: 1 2024-09-22T02:58:02.529 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T02:58:02.529 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T02:58:02.529 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-09-22 02:57:54.382057060 +0000 2024-09-22T02:58:02.529 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-09-22 02:54:36.793010965 +0000 2024-09-22T02:58:02.529 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-09-22 02:54:36.793010965 +0000 2024-09-22T02:58:02.529 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-09-22 02:54:36.793010965 +0000 2024-09-22T02:58:02.529 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T02:58:02.591 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-09-22T02:58:02.591 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-09-22T02:58:02.591 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000193229 s, 2.6 MB/s 2024-09-22T02:58:02.592 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T02:58:02.648 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-22T02:58:02.648 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:02.648 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T02:58:02.666 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T02:58:02.666 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-09-22T02:58:02.722 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T02:58:02.722 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T02:58:02.722 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-22T02:58:02.722 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T02:58:02.723 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T02:58:02.723 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-22 02:57:57.025557014 +0000 2024-09-22T02:58:02.723 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-22 02:54:37.688860236 +0000 2024-09-22T02:58:02.723 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-22 02:54:37.688860236 +0000 2024-09-22T02:58:02.723 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-22 02:54:37.688860236 +0000 2024-09-22T02:58:02.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T02:58:02.788 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-22T02:58:02.788 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-22T02:58:02.788 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000296138 s, 1.7 MB/s 2024-09-22T02:58:02.789 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T02:58:02.847 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-09-22T02:58:02.903 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T02:58:02.903 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T02:58:02.904 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-22T02:58:02.904 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T02:58:02.904 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T02:58:02.904 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-22 02:57:57.025557014 +0000 2024-09-22T02:58:02.904 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-22 02:54:38.011851359 +0000 2024-09-22T02:58:02.904 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-22 02:54:38.011851359 +0000 2024-09-22T02:58:02.904 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-22 02:54:38.011851359 +0000 2024-09-22T02:58:02.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T02:58:02.968 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-22T02:58:02.968 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-22T02:58:02.968 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000213388 s, 2.4 MB/s 2024-09-22T02:58:02.969 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T02:58:03.026 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-09-22T02:58:03.082 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T02:58:03.082 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T02:58:03.082 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-22T02:58:03.082 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T02:58:03.082 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T02:58:03.082 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-22 02:57:57.025557014 +0000 2024-09-22T02:58:03.082 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-22 02:54:38.326842701 +0000 2024-09-22T02:58:03.082 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-22 02:54:38.326842701 +0000 2024-09-22T02:58:03.082 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-22 02:54:38.326842701 +0000 2024-09-22T02:58:03.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T02:58:03.145 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-22T02:58:03.145 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-22T02:58:03.145 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000198287 s, 2.6 MB/s 2024-09-22T02:58:03.147 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T02:58:03.203 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-09-22T02:58:03.257 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T02:58:03.257 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T02:58:03.257 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-22T02:58:03.258 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T02:58:03.258 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T02:58:03.258 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-22 02:57:57.026556987 +0000 2024-09-22T02:58:03.258 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-22 02:54:38.642834017 +0000 2024-09-22T02:58:03.258 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-22 02:54:38.642834017 +0000 2024-09-22T02:58:03.258 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-09-22 02:54:38.642834017 +0000 2024-09-22T02:58:03.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T02:58:03.320 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-22T02:58:03.320 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-22T02:58:03.321 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000189958 s, 2.7 MB/s 2024-09-22T02:58:03.321 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T02:58:03.378 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-22T02:58:03.378 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-09-22T02:58:03.378 INFO:tasks.ceph:Generating config... 2024-09-22T02:58:03.379 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-22T02:58:03.379 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-22T02:58:03.379 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-22T02:58:03.379 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-22T02:58:03.379 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-22T02:58:03.379 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-09-22T02:58:03.379 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-22T02:58:03.379 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-22T02:58:03.379 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] err to stderr = True 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] log flush on exit = True 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] log to file = False 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] log to stderr = True 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-22T02:58:03.380 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-22T02:58:03.381 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-22T02:58:03.381 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-22T02:58:03.381 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-22T02:58:03.381 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-22T02:58:03.381 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-22T02:58:03.381 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-09-22T02:58:03.381 INFO:tasks.ceph:[client.0] admin socket = /var/run/ceph/ceph-$name.asok 2024-09-22T02:58:03.381 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-22T02:58:03.381 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-22T02:58:03.381 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-22T02:58:03.381 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-22T02:58:03.381 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-22T02:58:03.381 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-22T02:58:03.381 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-22T02:58:03.381 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-09-22T02:58:03.381 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-09-22T02:58:03.382 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-22T02:58:03.382 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-22T02:58:03.382 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-22T02:58:03.382 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-22T02:58:03.382 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-22T02:58:03.382 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-22T02:58:03.382 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-22T02:58:03.382 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-22T02:58:03.382 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-22T02:58:03.382 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-22T02:58:03.382 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-22T02:58:03.382 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-22T02:58:03.382 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-22T02:58:03.382 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-22T02:58:03.382 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-22T02:58:03.383 INFO:tasks.ceph:Setting up mon.a... 2024-09-22T02:58:03.383 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-22T02:58:03.451 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.keyring 2024-09-22T02:58:03.454 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-22T02:58:03.504 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T02:58:03.528 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.81')] 2024-09-22T02:58:03.529 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.81', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-22T02:58:03.529 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:03.530 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T02:58:03.583 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.81 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-22T02:58:03.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:03.668+0000 7f07fde92e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:03.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:03.669+0000 7f07fde92e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:03.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:03.669+0000 7f07fde92e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:03.671 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-22T02:58:03.671 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: generated fsid ba6605a2-3158-4c2e-9364-faeb0f665984 2024-09-22T02:58:03.671 INFO:teuthology.orchestra.run.smithi081.stdout:setting min_mon_release = quincy 2024-09-22T02:58:03.671 INFO:teuthology.orchestra.run.smithi081.stdout:epoch 0 2024-09-22T02:58:03.672 INFO:teuthology.orchestra.run.smithi081.stdout:fsid ba6605a2-3158-4c2e-9364-faeb0f665984 2024-09-22T02:58:03.672 INFO:teuthology.orchestra.run.smithi081.stdout:last_changed 2024-09-22T02:58:03.670332+0000 2024-09-22T02:58:03.672 INFO:teuthology.orchestra.run.smithi081.stdout:created 2024-09-22T02:58:03.670332+0000 2024-09-22T02:58:03.672 INFO:teuthology.orchestra.run.smithi081.stdout:min_mon_release 17 (quincy) 2024-09-22T02:58:03.672 INFO:teuthology.orchestra.run.smithi081.stdout:election_strategy: 1 2024-09-22T02:58:03.672 INFO:teuthology.orchestra.run.smithi081.stdout:0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-09-22T02:58:03.672 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-22T02:58:03.673 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T02:58:03.688 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ba6605a2-3158-4c2e-9364-faeb0f665984... 2024-09-22T02:58:03.689 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T02:58:03.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T02:58:03.770 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2024-09-22T02:58:03.770 INFO:teuthology.orchestra.run.smithi081.stdout: chdir = "" 2024-09-22T02:58:03.770 INFO:teuthology.orchestra.run.smithi081.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T02:58:03.770 INFO:teuthology.orchestra.run.smithi081.stdout: auth supported = cephx 2024-09-22T02:58:03.770 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.770 INFO:teuthology.orchestra.run.smithi081.stdout: filestore xattr use omap = true 2024-09-22T02:58:03.770 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: mon clock drift allowed = 1.000 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: osd crush chooseleaf type = 0 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: auth debug = true 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on old message = true 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on bug = true 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: mon pg warn max object skew = 0 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T02:58:03.771 INFO:teuthology.orchestra.run.smithi081.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default size = 2 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow primary affinity = true 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow pg remap = true 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on legacy crush tunables = false 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on crush straw calc version zero = false 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on no sortbitwise = false 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on osd down out interval zero = false 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on too few osds = false 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: mon_allow_pool_size_one = true 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.772 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: osd default data pool replay window = 5 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: mon allow pool delete = true 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: mon cluster log file level = debug 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: debug asserts on shutdown = true 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: mon health detail to clog = false 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: mon host = 172.21.15.81 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: ms client mode = crc 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: ms cluster mode = crc 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: ms mon client mode = crc 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: ms mon cluster mode = crc 2024-09-22T02:58:03.773 INFO:teuthology.orchestra.run.smithi081.stdout: ms mon service mode = crc 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: ms service mode = crc 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: osd_pool_default_min_size = 2 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: osd_pool_default_size = 3 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = ba6605a2-3158-4c2e-9364-faeb0f665984 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout:[osd] 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: osd journal size = 100 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub load threshold = 5.0 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub max interval = 120 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock profile = high_recovery_ops 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: osd recover clone overlap = true 2024-09-22T02:58:03.774 INFO:teuthology.orchestra.run.smithi081.stdout: osd recovery max chunk = 1048576 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug shutdown = true 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug op order = true 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify stray on activate = true 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug trim objects = true 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: osd open classes on start = true 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug pg log writeout = true 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: osd deep scrub update digest min age = 30 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: osd map max advance = 10 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: journal zero on create = true 2024-09-22T02:58:03.775 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: filestore ondisk finisher threads = 3 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: filestore apply finisher threads = 3 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: bdev debug aio = true 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug misdirected ops = true 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: debug monc = 20 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore block size = 96636764160 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore compression algorithm = snappy 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore compression mode = aggressive 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore fsck on mount = True 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore rocksdb cf = False 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: crimson osd obc lru size = 10 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluefs = 20 2024-09-22T02:58:03.776 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluestore = 20 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: debug osd = 20 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: debug rocksdb = 10 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: err to stderr = True 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: log flush on exit = True 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: log to file = False 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: log to stderr = True 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd backfillfull_ratio = 0.85 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd full ratio = 0.9 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd nearfull ratio = 0.8 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: osd blocked scrub grace period = 3600 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug reject backfill probability = 0.3 2024-09-22T02:58:03.777 INFO:teuthology.orchestra.run.smithi081.stdout: osd delete sleep = 1 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: osd failsafe full ratio = 0.95 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: osd max backfills = 3 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: osd max markdown count = 1000 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: osd objectstore = bluestore 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub min interval = 60 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: osd shutdown pgref assert = True 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: osd snap trim sleep = 2 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr] 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: debug mgr = 20 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-09-22T02:58:03.778 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: mgr/telemetry/nag = false 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout:[mon] 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: debug paxos = 20 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: mon data avail warn = 5 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: mon mgr mkfs grace = 240 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd reporter subtree level = osd 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd prime pg temp = true 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-09-22T02:58:03.779 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: # 1m isn't quite enough 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: mon_down_mkfs_grace = 2m 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.780 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_filestore_osds = false 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: mon min osdmap epochs = 50 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune interval = 2 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune min = 15 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune txsize = 2 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default crimson = True 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: paxos service trim min = 10 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout:[client] 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: rgw cache enabled = true 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable ops log = true 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable usage log = true 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.a] 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-09-22T02:58:03.781 INFO:teuthology.orchestra.run.smithi081.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-09-22T02:58:03.782 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.0] 2024-09-22T02:58:03.782 INFO:teuthology.orchestra.run.smithi081.stdout: crimson seastar cpu cores = 0-2 2024-09-22T02:58:03.782 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.1] 2024-09-22T02:58:03.782 INFO:teuthology.orchestra.run.smithi081.stdout: crimson seastar cpu cores = 3-5 2024-09-22T02:58:03.782 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.2] 2024-09-22T02:58:03.782 INFO:teuthology.orchestra.run.smithi081.stdout: crimson seastar cpu cores = 0-2 2024-09-22T02:58:03.782 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.3] 2024-09-22T02:58:03.782 INFO:teuthology.orchestra.run.smithi081.stdout: crimson seastar cpu cores = 3-5 2024-09-22T02:58:03.782 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.783 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T02:58:03.784 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = 172.21.15.81 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T02:58:03.785 INFO:teuthology.orchestra.run.smithi190.stdout: ms client mode = crc 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: ms cluster mode = crc 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: ms mon client mode = crc 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: ms mon cluster mode = crc 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: ms mon service mode = crc 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: ms service mode = crc 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_min_size = 2 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_size = 3 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = ba6605a2-3158-4c2e-9364-faeb0f665984 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 120 2024-09-22T02:58:03.786 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug trim objects = true 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.787 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: debug monc = 20 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore compression algorithm = snappy 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore compression mode = aggressive 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore rocksdb cf = False 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: crimson osd obc lru size = 10 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 20 2024-09-22T02:58:03.788 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 20 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 10 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: err to stderr = True 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: log flush on exit = True 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: log to file = False 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: log to stderr = True 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: osd blocked scrub grace period = 3600 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug reject backfill probability = 0.3 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: osd delete sleep = 1 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: osd max backfills = 3 2024-09-22T02:58:03.789 INFO:teuthology.orchestra.run.smithi190.stdout: osd max markdown count = 1000 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub min interval = 60 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: osd shutdown pgref assert = True 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: osd snap trim sleep = 2 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2024-09-22T02:58:03.790 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T02:58:03.791 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: mon min osdmap epochs = 50 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune interval = 2 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune min = 15 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune txsize = 2 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default crimson = True 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: paxos service trim min = 10 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-22T02:58:03.792 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: crimson seastar cpu cores = 0-2 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: crimson seastar cpu cores = 3-5 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: crimson seastar cpu cores = 0-2 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.3] 2024-09-22T02:58:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: crimson seastar cpu cores = 3-5 2024-09-22T02:58:03.795 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-22T02:58:03.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-22T02:58:03.894 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-22T02:58:03.894 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:03.894 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-22T02:58:03.909 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:03.910 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-22T02:58:03.964 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi081.front.sepia.ceph.com 2024-09-22T02:58:03.964 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:03.964 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T02:58:03.964 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T02:58:04.040 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:04.040 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T02:58:04.097 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2024-09-22T02:58:04.097 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:04.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T02:58:04.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T02:58:04.134 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:04.134 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T02:58:04.190 INFO:tasks.ceph:Setting up mon nodes... 2024-09-22T02:58:04.190 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-22T02:58:04.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T02:58:04.330 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T02:58:04.332 INFO:tasks.ceph:Setting up mds nodes... 2024-09-22T02:58:04.332 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-22T02:58:04.332 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-22T02:58:04.380 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-22T02:58:04.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-22T02:58:04.440 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-22T02:58:04.452 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-22T02:58:04.452 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-09-22T02:58:04.453 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-22T02:58:04.477 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-22T02:58:04.478 INFO:tasks.ceph:role: osd.0 2024-09-22T02:58:04.478 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com 2024-09-22T02:58:04.478 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T02:58:04.551 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T02:58:04.551 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T02:58:04.551 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T02:58:04.551 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T02:58:04.551 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T02:58:04.551 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-09-22T02:58:04.551 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T02:58:04.551 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T02:58:04.551 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T02:58:04.551 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T02:58:05.058 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-09-22T02:58:05.061 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-09-22T02:58:05.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-22T02:58:05.181 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-22T02:58:05.212 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-22T02:58:05.276 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-22T02:58:05.276 INFO:tasks.ceph:role: osd.1 2024-09-22T02:58:05.276 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com 2024-09-22T02:58:05.276 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T02:58:05.348 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T02:58:05.348 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T02:58:05.348 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T02:58:05.348 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T02:58:05.348 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T02:58:05.348 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-09-22T02:58:05.349 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T02:58:05.349 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T02:58:05.349 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T02:58:05.349 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T02:58:05.814 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-09-22T02:58:05.816 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-09-22T02:58:05.816 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-22T02:58:05.908 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-22T02:58:05.936 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T02:58:06.485 INFO:teuthology.orchestra.run.smithi081.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T02:58:06.487 INFO:teuthology.orchestra.run.smithi081.stderr:INFO 2024-09-22 02:58:06,486 seastar - Reactor backend: linux-aio 2024-09-22T02:58:06.508 INFO:teuthology.orchestra.run.smithi081.stderr:==27461==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T02:58:06.552 INFO:teuthology.orchestra.run.smithi081.stderr:INFO 2024-09-22 02:58:06,551 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T02:58:06.552 INFO:teuthology.orchestra.run.smithi081.stderr:INFO 2024-09-22 02:58:06,551 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-09-22T02:58:06.553 INFO:teuthology.orchestra.run.smithi081.stderr:DEBUG 2024-09-22 02:58:06,552 [shard 0:main] seastar - reactor::drain 2024-09-22T02:58:07.042 INFO:teuthology.orchestra.run.smithi081.stderr:----------------------------------------------------- 2024-09-22T02:58:07.042 INFO:teuthology.orchestra.run.smithi081.stderr:Suppressions used: 2024-09-22T02:58:07.042 INFO:teuthology.orchestra.run.smithi081.stderr: count bytes template 2024-09-22T02:58:07.042 INFO:teuthology.orchestra.run.smithi081.stderr: 1 8 MallocExtension::Register 2024-09-22T02:58:07.042 INFO:teuthology.orchestra.run.smithi081.stderr:----------------------------------------------------- 2024-09-22T02:58:07.043 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-09-22T02:58:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T02:58:07.100 INFO:teuthology.orchestra.run.smithi081.stderr:INFO 2024-09-22 02:58:07,099 seastar - Reactor backend: linux-aio 2024-09-22T02:58:07.127 INFO:teuthology.orchestra.run.smithi081.stderr:==27459==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T02:58:12.202 INFO:teuthology.orchestra.run.smithi081.stderr:----------------------------------------------------- 2024-09-22T02:58:12.202 INFO:teuthology.orchestra.run.smithi081.stderr:Suppressions used: 2024-09-22T02:58:12.202 INFO:teuthology.orchestra.run.smithi081.stderr: count bytes template 2024-09-22T02:58:12.202 INFO:teuthology.orchestra.run.smithi081.stderr: 1 8 MallocExtension::Register 2024-09-22T02:58:12.202 INFO:teuthology.orchestra.run.smithi081.stderr:----------------------------------------------------- 2024-09-22T02:58:12.202 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-09-22T02:58:12.202 INFO:teuthology.orchestra.run.smithi081.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid ba6605a2-3158-4c2e-9364-faeb0f665984 2024-09-22T02:58:12.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-22T02:58:12.266 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T02:58:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T02:58:12.809 INFO:teuthology.orchestra.run.smithi081.stderr:INFO 2024-09-22 02:58:12,808 seastar - Reactor backend: linux-aio 2024-09-22T02:58:12.831 INFO:teuthology.orchestra.run.smithi081.stderr:==27647==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T02:58:12.876 INFO:teuthology.orchestra.run.smithi081.stderr:INFO 2024-09-22 02:58:12,875 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T02:58:12.876 INFO:teuthology.orchestra.run.smithi081.stderr:INFO 2024-09-22 02:58:12,875 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-09-22T02:58:12.877 INFO:teuthology.orchestra.run.smithi081.stderr:DEBUG 2024-09-22 02:58:12,876 [shard 0:main] seastar - reactor::drain 2024-09-22T02:58:13.362 INFO:teuthology.orchestra.run.smithi081.stderr:----------------------------------------------------- 2024-09-22T02:58:13.362 INFO:teuthology.orchestra.run.smithi081.stderr:Suppressions used: 2024-09-22T02:58:13.362 INFO:teuthology.orchestra.run.smithi081.stderr: count bytes template 2024-09-22T02:58:13.362 INFO:teuthology.orchestra.run.smithi081.stderr: 1 8 MallocExtension::Register 2024-09-22T02:58:13.362 INFO:teuthology.orchestra.run.smithi081.stderr:----------------------------------------------------- 2024-09-22T02:58:13.363 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-09-22T02:58:13.421 INFO:teuthology.orchestra.run.smithi081.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T02:58:13.422 INFO:teuthology.orchestra.run.smithi081.stderr:INFO 2024-09-22 02:58:13,422 seastar - Reactor backend: linux-aio 2024-09-22T02:58:13.450 INFO:teuthology.orchestra.run.smithi081.stderr:==27645==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T02:58:18.565 INFO:teuthology.orchestra.run.smithi081.stderr:----------------------------------------------------- 2024-09-22T02:58:18.565 INFO:teuthology.orchestra.run.smithi081.stderr:Suppressions used: 2024-09-22T02:58:18.565 INFO:teuthology.orchestra.run.smithi081.stderr: count bytes template 2024-09-22T02:58:18.566 INFO:teuthology.orchestra.run.smithi081.stderr: 1 8 MallocExtension::Register 2024-09-22T02:58:18.566 INFO:teuthology.orchestra.run.smithi081.stderr:----------------------------------------------------- 2024-09-22T02:58:18.566 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-09-22T02:58:18.566 INFO:teuthology.orchestra.run.smithi081.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid ba6605a2-3158-4c2e-9364-faeb0f665984 2024-09-22T02:58:18.605 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-22T02:58:18.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-22T02:58:18.660 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-22T02:58:18.660 INFO:tasks.ceph:role: osd.2 2024-09-22T02:58:18.660 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-22T02:58:18.660 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T02:58:18.737 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T02:58:18.737 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T02:58:18.737 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T02:58:18.737 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T02:58:18.737 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T02:58:18.737 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-22T02:58:18.737 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T02:58:18.737 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T02:58:18.737 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T02:58:18.737 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T02:58:19.252 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-22T02:58:19.254 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-22T02:58:19.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-22T02:58:19.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-22T02:58:19.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-22T02:58:19.515 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-22T02:58:19.516 INFO:tasks.ceph:role: osd.3 2024-09-22T02:58:19.516 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-22T02:58:19.516 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T02:58:19.590 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T02:58:19.590 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T02:58:19.590 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T02:58:19.590 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T02:58:19.590 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T02:58:19.590 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-22T02:58:19.590 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T02:58:19.590 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T02:58:19.590 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T02:58:19.590 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T02:58:20.040 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-22T02:58:20.043 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-22T02:58:20.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-09-22T02:58:20.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-22T02:58:20.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T02:58:20.753 INFO:teuthology.orchestra.run.smithi190.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T02:58:20.754 INFO:teuthology.orchestra.run.smithi190.stderr:INFO 2024-09-22 02:58:20,753 seastar - Reactor backend: linux-aio 2024-09-22T02:58:20.772 INFO:teuthology.orchestra.run.smithi190.stderr:==27633==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T02:58:20.820 INFO:teuthology.orchestra.run.smithi190.stderr:INFO 2024-09-22 02:58:20,819 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T02:58:20.820 INFO:teuthology.orchestra.run.smithi190.stderr:INFO 2024-09-22 02:58:20,819 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-09-22T02:58:20.821 INFO:teuthology.orchestra.run.smithi190.stderr:DEBUG 2024-09-22 02:58:20,820 [shard 0:main] seastar - reactor::drain 2024-09-22T02:58:21.320 INFO:teuthology.orchestra.run.smithi190.stderr:----------------------------------------------------- 2024-09-22T02:58:21.320 INFO:teuthology.orchestra.run.smithi190.stderr:Suppressions used: 2024-09-22T02:58:21.321 INFO:teuthology.orchestra.run.smithi190.stderr: count bytes template 2024-09-22T02:58:21.321 INFO:teuthology.orchestra.run.smithi190.stderr: 1 8 MallocExtension::Register 2024-09-22T02:58:21.321 INFO:teuthology.orchestra.run.smithi190.stderr:----------------------------------------------------- 2024-09-22T02:58:21.321 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-22T02:58:21.373 INFO:teuthology.orchestra.run.smithi190.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T02:58:21.375 INFO:teuthology.orchestra.run.smithi190.stderr:INFO 2024-09-22 02:58:21,374 seastar - Reactor backend: linux-aio 2024-09-22T02:58:21.412 INFO:teuthology.orchestra.run.smithi190.stderr:==27631==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T02:58:26.495 INFO:teuthology.orchestra.run.smithi190.stderr:----------------------------------------------------- 2024-09-22T02:58:26.495 INFO:teuthology.orchestra.run.smithi190.stderr:Suppressions used: 2024-09-22T02:58:26.495 INFO:teuthology.orchestra.run.smithi190.stderr: count bytes template 2024-09-22T02:58:26.495 INFO:teuthology.orchestra.run.smithi190.stderr: 1 8 MallocExtension::Register 2024-09-22T02:58:26.495 INFO:teuthology.orchestra.run.smithi190.stderr:----------------------------------------------------- 2024-09-22T02:58:26.495 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-22T02:58:26.495 INFO:teuthology.orchestra.run.smithi190.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid ba6605a2-3158-4c2e-9364-faeb0f665984 2024-09-22T02:58:26.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-22T02:58:26.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T02:58:27.109 INFO:teuthology.orchestra.run.smithi190.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T02:58:27.110 INFO:teuthology.orchestra.run.smithi190.stderr:INFO 2024-09-22 02:58:27,109 seastar - Reactor backend: linux-aio 2024-09-22T02:58:27.132 INFO:teuthology.orchestra.run.smithi190.stderr:==27820==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T02:58:27.178 INFO:teuthology.orchestra.run.smithi190.stderr:INFO 2024-09-22 02:58:27,177 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T02:58:27.178 INFO:teuthology.orchestra.run.smithi190.stderr:INFO 2024-09-22 02:58:27,177 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-09-22T02:58:27.178 INFO:teuthology.orchestra.run.smithi190.stderr:DEBUG 2024-09-22 02:58:27,177 [shard 0:main] seastar - reactor::drain 2024-09-22T02:58:27.659 INFO:teuthology.orchestra.run.smithi190.stderr:----------------------------------------------------- 2024-09-22T02:58:27.659 INFO:teuthology.orchestra.run.smithi190.stderr:Suppressions used: 2024-09-22T02:58:27.659 INFO:teuthology.orchestra.run.smithi190.stderr: count bytes template 2024-09-22T02:58:27.659 INFO:teuthology.orchestra.run.smithi190.stderr: 1 8 MallocExtension::Register 2024-09-22T02:58:27.659 INFO:teuthology.orchestra.run.smithi190.stderr:----------------------------------------------------- 2024-09-22T02:58:27.659 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-22T02:58:27.717 INFO:teuthology.orchestra.run.smithi190.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T02:58:27.719 INFO:teuthology.orchestra.run.smithi190.stderr:INFO 2024-09-22 02:58:27,718 seastar - Reactor backend: linux-aio 2024-09-22T02:58:27.754 INFO:teuthology.orchestra.run.smithi190.stderr:==27818==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T02:58:32.849 INFO:teuthology.orchestra.run.smithi190.stderr:----------------------------------------------------- 2024-09-22T02:58:32.849 INFO:teuthology.orchestra.run.smithi190.stderr:Suppressions used: 2024-09-22T02:58:32.849 INFO:teuthology.orchestra.run.smithi190.stderr: count bytes template 2024-09-22T02:58:32.849 INFO:teuthology.orchestra.run.smithi190.stderr: 1 8 MallocExtension::Register 2024-09-22T02:58:32.850 INFO:teuthology.orchestra.run.smithi190.stderr:----------------------------------------------------- 2024-09-22T02:58:32.850 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-22T02:58:32.850 INFO:teuthology.orchestra.run.smithi190.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid ba6605a2-3158-4c2e-9364-faeb0f665984 2024-09-22T02:58:32.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-22T02:58:32.913 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-22T02:58:32.913 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:32.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-22T02:58:32.944 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:32.944 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-22T02:58:33.013 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:33.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-22T02:58:33.038 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:33.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-22T02:58:33.105 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:33.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-22T02:58:33.169 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:33.169 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-22T02:58:33.187 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:33.188 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-22T02:58:33.225 INFO:tasks.ceph:Adding keys to all mons... 2024-09-22T02:58:33.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-22T02:58:33.254 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.0] 2024-09-22T02:58:33.254 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC/h+9mx00HCxAAle1RTAaGZNWwIQ8C0QHQ0g== 2024-09-22T02:58:33.254 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.1] 2024-09-22T02:58:33.254 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDFh+9mt5tEHhAAZt5FN4eOwvV9kuWIyZDHDA== 2024-09-22T02:58:33.254 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.x] 2024-09-22T02:58:33.254 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC8h+9mxnyTExAAVT0FvRlVVv0A5WN+lJ7yow== 2024-09-22T02:58:33.254 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.2] 2024-09-22T02:58:33.254 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDNh+9miwIAHBAAo8htQ9OFkQROoMSXeqGxNA== 2024-09-22T02:58:33.254 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.3] 2024-09-22T02:58:33.254 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDTh+9mUlRjMBAA4k/YFKdhtVXAnFq3bkV3aw== 2024-09-22T02:58:33.255 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-09-22T02:58:33.255 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC8h+9mJC+gFhAAYqgt3MBWq2L2UK+oY06YMQ== 2024-09-22T02:58:33.255 INFO:teuthology.orchestra.run.smithi081.stdout:[client.1] 2024-09-22T02:58:33.255 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC8h+9m9zcwGhAAZxmweZXntJ88Koq8qbfmQQ== 2024-09-22T02:58:33.256 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T02:58:33.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T02:58:33.418 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-22T02:58:33.475 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T02:58:33.531 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T02:58:33.588 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T02:58:33.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T02:58:33.706 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-22T02:58:33.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-22T02:58:33.731 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-22T02:58:33.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:33.834+0000 7fec18a61b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:33.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:33.834+0000 7fec18a61b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:33.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-22T02:58:33.873 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-22T02:58:33.928 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-22T02:58:33.929 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-22T02:58:33.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-22T02:58:33.971 INFO:tasks.ceph.mon.a:Started 2024-09-22T02:58:33.971 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-22T02:58:33.971 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-22T02:58:33.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-22T02:58:33.974 INFO:tasks.ceph.mgr.x:Started 2024-09-22T02:58:33.974 INFO:tasks.ceph:Setting crush tunables to default 2024-09-22T02:58:33.974 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd crush tunables default 2024-09-22T02:58:34.048 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-09-22T02:58:34.046+0000 7f419b42fb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:34.048 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-09-22T02:58:34.046+0000 7f419b42fb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:34.048 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-09-22T02:58:34.046+0000 7f419b42fb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:34.073 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:34.071+0000 7fae3c3c0f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:34.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:34.071+0000 7f69db82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:34.091 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:34.090+0000 7fae3c3c0f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:34.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:34.090+0000 7f69db82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:34.092 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:34.090+0000 7fae3c3c0f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:34.159 INFO:teuthology.orchestra.run.smithi081.stderr:adjusted tunables profile to default 2024-09-22T02:58:34.170 INFO:tasks.ceph:check_enable_crimson: True 2024-09-22T02:58:34.170 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-22T02:58:34.170 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-22T02:58:34.189 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:34.188+0000 7fae3c3c0f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-22T02:58:34.230 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:34.228+0000 7fae3c3c0f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-22T02:58:34.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:34.264+0000 7f8dac757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:34.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:34.272+0000 7f8dac757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:34.349 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-09-22T02:58:34.347+0000 7f41933f1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-22T02:58:34.349 INFO:tasks.ceph.mon.a.smithi081.stderr:Please be aware that this feature is experimental, untested, 2024-09-22T02:58:34.349 INFO:tasks.ceph.mon.a.smithi081.stderr:unsupported, and may result in data corruption, data loss, 2024-09-22T02:58:34.349 INFO:tasks.ceph.mon.a.smithi081.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-22T02:58:34.349 INFO:tasks.ceph.mon.a.smithi081.stderr:feature with important data. 2024-09-22T02:58:34.349 INFO:tasks.ceph.mon.a.smithi081.stderr: 2024-09-22T02:58:34.367 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:34.365+0000 7fae3c3c0f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-22T02:58:34.417 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:34.415+0000 7fae3c3c0f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-22T02:58:34.571 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:34.569+0000 7fae3c3c0f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-22T02:58:34.679 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:34.677+0000 7fae3c3c0f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-22T02:58:35.002 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:35.000+0000 7fae3c3c0f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-22T02:58:35.057 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:35.055+0000 7fae3c3c0f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-22T02:58:35.172 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-22T02:58:35.173 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:35.173 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-22T02:58:35.199 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-09-22T02:58:35.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-22T02:58:35.205 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:35.203+0000 7fae3c3c0f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-22T02:58:35.265 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:35.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-22T02:58:35.291 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-22T02:58:35.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-22T02:58:35.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new d8dc2c56-1fa8-4c94-b1c4-1ff6bb0919b5 0 2024-09-22T02:58:35.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-22T02:58:35.494+0000 7fddee797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:35.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-22T02:58:35.502+0000 7fddee797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:35.581 INFO:teuthology.orchestra.run.smithi190.stdout:0 2024-09-22T02:58:35.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 4c95c86d-0d11-4418-921f-c13d799252d2 1 2024-09-22T02:58:35.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-22T02:58:35.689+0000 7f4057797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:35.699 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-22T02:58:35.697+0000 7f4057797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:35.775 INFO:teuthology.orchestra.run.smithi190.stdout:1 2024-09-22T02:58:35.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new c70b3bc0-7a70-4652-a939-acac6432f224 2 2024-09-22T02:58:35.891 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-22T02:58:35.883+0000 7f83114ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:35.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-22T02:58:35.892+0000 7f83114ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:35.968 INFO:teuthology.orchestra.run.smithi190.stdout:2 2024-09-22T02:58:35.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 4047b1af-c441-4a8f-bc0d-7c2a4f237e18 3 2024-09-22T02:58:36.083 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-22T02:58:36.076+0000 7f73f24d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:36.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-22T02:58:36.084+0000 7f73f24d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:36.164 INFO:teuthology.orchestra.run.smithi190.stdout:3 2024-09-22T02:58:36.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-22T02:58:36.203 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:36.201+0000 7fae3c3c0f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-22T02:58:36.211 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T02:58:36.211 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T02:58:36.253 INFO:tasks.ceph.osd.0:Started 2024-09-22T02:58:36.253 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-22T02:58:36.256 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:36.253+0000 7fae3c3c0f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-22T02:58:36.285 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T02:58:36.285 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T02:58:36.328 INFO:tasks.ceph.osd.1:Started 2024-09-22T02:58:36.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-22T02:58:36.358 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T02:58:36.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T02:58:36.400 INFO:tasks.ceph.osd.2:Started 2024-09-22T02:58:36.400 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-09-22T02:58:36.428 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T02:58:36.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T02:58:36.472 INFO:tasks.ceph.osd.3:Started 2024-09-22T02:58:36.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T02:58:36.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:36.586+0000 7f1dc246c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:36.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:36.594+0000 7f1dc246c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:36.664 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:36.664 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":7,"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","created":"2024-09-22T02:58:34.078383+0000","modified":"2024-09-22T02:58:36.162126+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T02:58:36.162126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d8dc2c56-1fa8-4c94-b1c4-1ff6bb0919b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4c95c86d-0d11-4418-921f-c13d799252d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c70b3bc0-7a70-4652-a939-acac6432f224","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4047b1af-c441-4a8f-bc0d-7c2a4f237e18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T02:58:36.667 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:36.665+0000 7fae3c3c0f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-22T02:58:36.677 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-22T02:58:36.677 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-22T02:58:36.772 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:36.770+0000 7fae3c3c0f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-22T02:58:36.896 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:36.894+0000 7fae3c3c0f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-22T02:58:36.947 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:36.946+0000 7fae3c3c0f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-22T02:58:37.069 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:37.067+0000 7fae3c3c0f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-22T02:58:37.080 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T02:58:37.126 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:37.124+0000 7fae3c3c0f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-22T02:58:37.176 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:37.174+0000 7fae3c3c0f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-22T02:58:37.179 INFO:teuthology.misc.health.smithi081.stderr:2024-09-22T02:58:37.174+0000 7f6590f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:37.183 INFO:teuthology.misc.health.smithi081.stderr:2024-09-22T02:58:37.181+0000 7f6590f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:37.248 INFO:teuthology.misc.health.smithi081.stdout: 2024-09-22T02:58:37.248 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":7,"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","created":"2024-09-22T02:58:34.078383+0000","modified":"2024-09-22T02:58:36.162126+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T02:58:36.162126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d8dc2c56-1fa8-4c94-b1c4-1ff6bb0919b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4c95c86d-0d11-4418-921f-c13d799252d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c70b3bc0-7a70-4652-a939-acac6432f224","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4047b1af-c441-4a8f-bc0d-7c2a4f237e18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T02:58:37.258 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-22T02:58:37.299 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:37.297+0000 7fae3c3c0f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-22T02:58:37.353 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:37.352+0000 7fae3c3c0f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-22T02:58:37.409 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:37.408+0000 7fae3c3c0f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-22T02:58:37.484 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:37.482+0000 7fae3c3c0f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-22T02:58:37.532 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:37.531+0000 7fae3c3c0f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-22T02:58:37.581 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:37.579+0000 7fae3c3c0f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-22T02:58:38.279 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:38.277+0000 7fae3c3c0f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-22T02:58:39.290 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-22T02:58:39.288+0000 7faddab65640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-22T02:58:43.662 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T02:58:43.769 INFO:teuthology.misc.health.smithi081.stderr:2024-09-22T02:58:43.764+0000 7f3a0df17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:43.772 INFO:teuthology.misc.health.smithi081.stderr:2024-09-22T02:58:43.771+0000 7f3a0df17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:43.950 INFO:teuthology.misc.health.smithi081.stdout: 2024-09-22T02:58:43.950 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","created":"2024-09-22T02:58:34.078383+0000","modified":"2024-09-22T02:58:42.295402+0000","last_up_change":"2024-09-22T02:58:42.295402+0000","last_in_change":"2024-09-22T02:58:36.162126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d8dc2c56-1fa8-4c94-b1c4-1ff6bb0919b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3135721300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6805","nonce":3135721300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3135721300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3135721300}]},"public_addr":"172.21.15.81:6804/3135721300","cluster_addr":"172.21.15.81:6805/3135721300","heartbeat_back_addr":"172.21.15.81:6807/3135721300","heartbeat_front_addr":"172.21.15.81:6806/3135721300","state":["exists","up"]},{"osd":1,"uuid":"4c95c86d-0d11-4418-921f-c13d799252d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3974603893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6803","nonce":3974603893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3974603893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6801","nonce":3974603893}]},"public_addr":"172.21.15.81:6802/3974603893","cluster_addr":"172.21.15.81:6803/3974603893","heartbeat_back_addr":"172.21.15.81:6800/3974603893","heartbeat_front_addr":"172.21.15.81:6801/3974603893","state":["exists","up"]},{"osd":2,"uuid":"c70b3bc0-7a70-4652-a939-acac6432f224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":294896312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":294896312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":294896312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":294896312}]},"public_addr":"172.21.15.190:6808/294896312","cluster_addr":"172.21.15.190:6809/294896312","heartbeat_back_addr":"172.21.15.190:6810/294896312","heartbeat_front_addr":"172.21.15.190:6807/294896312","state":["exists","up"]},{"osd":3,"uuid":"4047b1af-c441-4a8f-bc0d-7c2a4f237e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":216282597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":216282597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":216282597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":216282597}]},"public_addr":"172.21.15.190:6802/216282597","cluster_addr":"172.21.15.190:6803/216282597","heartbeat_back_addr":"172.21.15.190:6805/216282597","heartbeat_front_addr":"172.21.15.190:6806/216282597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T02:58:43.961 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-22T02:58:43.962 INFO:tasks.ceph:Creating RBD pool 2024-09-22T02:58:43.962 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-22T02:58:44.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:44.056+0000 7feea8e4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:44.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:44.066+0000 7feea8e4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:44.319 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'rbd' created 2024-09-22T02:58:44.346 DEBUG:teuthology.orchestra.run.smithi081:> rbd --cluster ceph pool init rbd 2024-09-22T02:58:44.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:44.408+0000 7fe44ce90d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:44.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:44.408+0000 7fe44ce90d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:44.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:44.412+0000 7fe44ce90d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:47.362 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-22T02:58:47.363 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-22T02:58:47.363 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-22T02:58:47.363 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-22T02:58:47.363 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-22T02:58:47.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:47.466+0000 7fd02446c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:47.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:47.472+0000 7fd02446c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:47.696 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:47.708 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2121483918},{"type":"v1","addr":"172.21.15.190:6801","nonce":2121483918}]},"active_addr":"172.21.15.190:6801/2121483918","active_change":"2024-09-22T02:58:38.279817+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":2295585775}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":2799926304}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":1919649151}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":360211972}]}]} 2024-09-22T02:58:47.710 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-22T02:58:47.710 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T02:58:47.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T02:58:47.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:47.808+0000 7f6d05797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:47.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:47.814+0000 7f6d05797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:47.994 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:47.994 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":14,"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","created":"2024-09-22T02:58:34.078383+0000","modified":"2024-09-22T02:58:47.323897+0000","last_up_change":"2024-09-22T02:58:42.295402+0000","last_in_change":"2024-09-22T02:58:36.162126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T02:58:44.262077+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T02:58:44.315438+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8dc2c56-1fa8-4c94-b1c4-1ff6bb0919b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3135721300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6805","nonce":3135721300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3135721300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3135721300}]},"public_addr":"172.21.15.81:6804/3135721300","cluster_addr":"172.21.15.81:6805/3135721300","heartbeat_back_addr":"172.21.15.81:6807/3135721300","heartbeat_front_addr":"172.21.15.81:6806/3135721300","state":["exists","up"]},{"osd":1,"uuid":"4c95c86d-0d11-4418-921f-c13d799252d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3974603893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6803","nonce":3974603893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3974603893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6801","nonce":3974603893}]},"public_addr":"172.21.15.81:6802/3974603893","cluster_addr":"172.21.15.81:6803/3974603893","heartbeat_back_addr":"172.21.15.81:6800/3974603893","heartbeat_front_addr":"172.21.15.81:6801/3974603893","state":["exists","up"]},{"osd":2,"uuid":"c70b3bc0-7a70-4652-a939-acac6432f224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":294896312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":294896312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":294896312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":294896312}]},"public_addr":"172.21.15.190:6808/294896312","cluster_addr":"172.21.15.190:6809/294896312","heartbeat_back_addr":"172.21.15.190:6810/294896312","heartbeat_front_addr":"172.21.15.190:6807/294896312","state":["exists","up"]},{"osd":3,"uuid":"4047b1af-c441-4a8f-bc0d-7c2a4f237e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":216282597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":216282597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":216282597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":216282597}]},"public_addr":"172.21.15.190:6802/216282597","cluster_addr":"172.21.15.190:6803/216282597","heartbeat_back_addr":"172.21.15.190:6805/216282597","heartbeat_front_addr":"172.21.15.190:6806/216282597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T02:58:48.007 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T02:58:48.007 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T02:58:48.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.105+0000 7f9eb730e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.111+0000 7f9eb730e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.292 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:48.292 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":14,"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","created":"2024-09-22T02:58:34.078383+0000","modified":"2024-09-22T02:58:47.323897+0000","last_up_change":"2024-09-22T02:58:42.295402+0000","last_in_change":"2024-09-22T02:58:36.162126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T02:58:44.262077+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T02:58:44.315438+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8dc2c56-1fa8-4c94-b1c4-1ff6bb0919b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3135721300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6805","nonce":3135721300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3135721300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3135721300}]},"public_addr":"172.21.15.81:6804/3135721300","cluster_addr":"172.21.15.81:6805/3135721300","heartbeat_back_addr":"172.21.15.81:6807/3135721300","heartbeat_front_addr":"172.21.15.81:6806/3135721300","state":["exists","up"]},{"osd":1,"uuid":"4c95c86d-0d11-4418-921f-c13d799252d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3974603893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6803","nonce":3974603893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3974603893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6801","nonce":3974603893}]},"public_addr":"172.21.15.81:6802/3974603893","cluster_addr":"172.21.15.81:6803/3974603893","heartbeat_back_addr":"172.21.15.81:6800/3974603893","heartbeat_front_addr":"172.21.15.81:6801/3974603893","state":["exists","up"]},{"osd":2,"uuid":"c70b3bc0-7a70-4652-a939-acac6432f224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":294896312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":294896312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":294896312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":294896312}]},"public_addr":"172.21.15.190:6808/294896312","cluster_addr":"172.21.15.190:6809/294896312","heartbeat_back_addr":"172.21.15.190:6810/294896312","heartbeat_front_addr":"172.21.15.190:6807/294896312","state":["exists","up"]},{"osd":3,"uuid":"4047b1af-c441-4a8f-bc0d-7c2a4f237e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":216282597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":216282597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":216282597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":216282597}]},"public_addr":"172.21.15.190:6802/216282597","cluster_addr":"172.21.15.190:6803/216282597","heartbeat_back_addr":"172.21.15.190:6805/216282597","heartbeat_front_addr":"172.21.15.190:6806/216282597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T02:58:48.305 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T02:58:48.305 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T02:58:48.306 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T02:58:48.306 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-22T02:58:48.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.414+0000 7f9ff9f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.414+0000 7efd82397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.414+0000 7f588cb17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.414+0000 7f6ad3863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.423+0000 7f588cb17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.423+0000 7efd82397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.424+0000 7f9ff9f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.424+0000 7f6ad3863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.455 INFO:teuthology.orchestra.run.smithi081.stdout:42949672966 2024-09-22T02:58:48.455 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-22T02:58:48.456 INFO:teuthology.orchestra.run.smithi081.stdout:42949672966 2024-09-22T02:58:48.457 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T02:58:48.458 INFO:teuthology.orchestra.run.smithi081.stdout:42949672966 2024-09-22T02:58:48.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T02:58:48.461 INFO:teuthology.orchestra.run.smithi081.stdout:38654705671 2024-09-22T02:58:48.462 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T02:58:48.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.558+0000 7fb0ea2d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.559+0000 7f7a2d597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.567+0000 7f7a2d597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.567+0000 7fb0ea2d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.605+0000 7ff6ca957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.605+0000 7fc007957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.613+0000 7ff6ca957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:48.614+0000 7fc007957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:48.759 INFO:teuthology.orchestra.run.smithi081.stdout:42949672966 2024-09-22T02:58:48.765 INFO:teuthology.orchestra.run.smithi081.stdout:42949672966 2024-09-22T02:58:48.773 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672966 for osd.0 2024-09-22T02:58:48.773 DEBUG:teuthology.parallel:result is None 2024-09-22T02:58:48.782 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672966 for osd.3 2024-09-22T02:58:48.782 DEBUG:teuthology.parallel:result is None 2024-09-22T02:58:48.813 INFO:teuthology.orchestra.run.smithi081.stdout:42949672966 2024-09-22T02:58:48.814 INFO:teuthology.orchestra.run.smithi081.stdout:38654705668 2024-09-22T02:58:48.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672966 for osd.2 2024-09-22T02:58:48.827 DEBUG:teuthology.parallel:result is None 2024-09-22T02:58:48.827 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705668 for osd.1 2024-09-22T02:58:49.828 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T02:58:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:49.929+0000 7fc81142c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:49.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:49.936+0000 7fc81142c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:50.118 INFO:teuthology.orchestra.run.smithi081.stdout:38654705668 2024-09-22T02:58:50.131 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705668 for osd.1 2024-09-22T02:58:51.132 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T02:58:51.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:51.238+0000 7f76cb117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:51.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:51.246+0000 7f76cb117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:51.429 INFO:teuthology.orchestra.run.smithi081.stdout:38654705671 2024-09-22T02:58:51.442 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.1 2024-09-22T02:58:51.442 DEBUG:teuthology.parallel:result is None 2024-09-22T02:58:51.442 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T02:58:51.442 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T02:58:51.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:51.540+0000 7f5375a2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:51.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:51.546+0000 7f5375a2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:51.723 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:51.723 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T02:58:51.735 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-22T02:58:50.286593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108020,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379340,"statfs":{"total":386547056640,"available":386436444160,"internally_reserved":0,"allocated":671744,"data_stored":1894113,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.841896"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346586+0000","last_change":"2024-09-22T02:58:47.347756+0000","last_active":"2024-09-22T02:58:47.346586+0000","last_peered":"2024-09-22T02:58:47.346586+0000","last_clean":"2024-09-22T02:58:47.346586+0000","last_became_active":"2024-09-22T02:58:45.389306+0000","last_became_peered":"2024-09-22T02:58:45.389306+0000","last_unstale":"2024-09-22T02:58:47.346586+0000","last_undegraded":"2024-09-22T02:58:47.346586+0000","last_fullsized":"2024-09-22T02:58:47.346586+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346635+0000","last_change":"2024-09-22T02:58:47.347783+0000","last_active":"2024-09-22T02:58:47.346635+0000","last_peered":"2024-09-22T02:58:47.346635+0000","last_clean":"2024-09-22T02:58:47.346635+0000","last_became_active":"2024-09-22T02:58:45.383906+0000","last_became_peered":"2024-09-22T02:58:45.383906+0000","last_unstale":"2024-09-22T02:58:47.346635+0000","last_undegraded":"2024-09-22T02:58:47.346635+0000","last_fullsized":"2024-09-22T02:58:47.346635+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346786+0000","last_change":"2024-09-22T02:58:47.347987+0000","last_active":"2024-09-22T02:58:47.346786+0000","last_peered":"2024-09-22T02:58:47.346786+0000","last_clean":"2024-09-22T02:58:47.346786+0000","last_became_active":"2024-09-22T02:58:45.364066+0000","last_became_peered":"2024-09-22T02:58:45.364066+0000","last_unstale":"2024-09-22T02:58:47.346786+0000","last_undegraded":"2024-09-22T02:58:47.346786+0000","last_fullsized":"2024-09-22T02:58:47.346786+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.344103+0000","last_change":"2024-09-22T02:58:47.346116+0000","last_active":"2024-09-22T02:58:47.344103+0000","last_peered":"2024-09-22T02:58:47.344103+0000","last_clean":"2024-09-22T02:58:47.344103+0000","last_became_active":"2024-09-22T02:58:45.374991+0000","last_became_peered":"2024-09-22T02:58:45.374991+0000","last_unstale":"2024-09-22T02:58:47.344103+0000","last_undegraded":"2024-09-22T02:58:47.344103+0000","last_fullsized":"2024-09-22T02:58:47.344103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.344867+0000","last_change":"2024-09-22T02:58:45.382298+0000","last_active":"2024-09-22T02:58:47.344867+0000","last_peered":"2024-09-22T02:58:47.344867+0000","last_clean":"2024-09-22T02:58:47.344867+0000","last_became_active":"2024-09-22T02:58:45.373807+0000","last_became_peered":"2024-09-22T02:58:45.373807+0000","last_unstale":"2024-09-22T02:58:47.344867+0000","last_undegraded":"2024-09-22T02:58:47.344867+0000","last_fullsized":"2024-09-22T02:58:47.344867+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346501+0000","last_change":"2024-09-22T02:58:47.348407+0000","last_active":"2024-09-22T02:58:47.346501+0000","last_peered":"2024-09-22T02:58:47.346501+0000","last_clean":"2024-09-22T02:58:47.346501+0000","last_became_active":"2024-09-22T02:58:45.369401+0000","last_became_peered":"2024-09-22T02:58:45.369401+0000","last_unstale":"2024-09-22T02:58:47.346501+0000","last_undegraded":"2024-09-22T02:58:47.346501+0000","last_fullsized":"2024-09-22T02:58:47.346501+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346629+0000","last_change":"2024-09-22T02:58:47.346629+0000","last_active":"2024-09-22T02:58:47.346629+0000","last_peered":"2024-09-22T02:58:47.346629+0000","last_clean":"2024-09-22T02:58:47.346629+0000","last_became_active":"2024-09-22T02:58:45.371859+0000","last_became_peered":"2024-09-22T02:58:45.371859+0000","last_unstale":"2024-09-22T02:58:47.346629+0000","last_undegraded":"2024-09-22T02:58:47.346629+0000","last_fullsized":"2024-09-22T02:58:47.346629+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.342743+0000","last_change":"2024-09-22T02:58:47.344427+0000","last_active":"2024-09-22T02:58:47.342743+0000","last_peered":"2024-09-22T02:58:47.342743+0000","last_clean":"2024-09-22T02:58:47.342743+0000","last_became_active":"2024-09-22T02:58:45.366840+0000","last_became_peered":"2024-09-22T02:58:45.366840+0000","last_unstale":"2024-09-22T02:58:47.342743+0000","last_undegraded":"2024-09-22T02:58:47.342743+0000","last_fullsized":"2024-09-22T02:58:47.342743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.343743+0000","last_change":"2024-09-22T02:58:47.345653+0000","last_active":"2024-09-22T02:58:47.343743+0000","last_peered":"2024-09-22T02:58:47.343743+0000","last_clean":"2024-09-22T02:58:47.343743+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T02:58:47.343743+0000","last_undegraded":"2024-09-22T02:58:47.343743+0000","last_fullsized":"2024-09-22T02:58:47.343743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":30757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":180224,"data_stored":621106,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T02:58:51.736 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T02:58:51.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:51.833+0000 7f7e14c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:51.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:51.839+0000 7f7e14c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:52.018 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:52.018 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T02:58:52.030 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-22T02:58:50.286593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108020,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379340,"statfs":{"total":386547056640,"available":386436444160,"internally_reserved":0,"allocated":671744,"data_stored":1894113,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.841896"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346586+0000","last_change":"2024-09-22T02:58:47.347756+0000","last_active":"2024-09-22T02:58:47.346586+0000","last_peered":"2024-09-22T02:58:47.346586+0000","last_clean":"2024-09-22T02:58:47.346586+0000","last_became_active":"2024-09-22T02:58:45.389306+0000","last_became_peered":"2024-09-22T02:58:45.389306+0000","last_unstale":"2024-09-22T02:58:47.346586+0000","last_undegraded":"2024-09-22T02:58:47.346586+0000","last_fullsized":"2024-09-22T02:58:47.346586+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346635+0000","last_change":"2024-09-22T02:58:47.347783+0000","last_active":"2024-09-22T02:58:47.346635+0000","last_peered":"2024-09-22T02:58:47.346635+0000","last_clean":"2024-09-22T02:58:47.346635+0000","last_became_active":"2024-09-22T02:58:45.383906+0000","last_became_peered":"2024-09-22T02:58:45.383906+0000","last_unstale":"2024-09-22T02:58:47.346635+0000","last_undegraded":"2024-09-22T02:58:47.346635+0000","last_fullsized":"2024-09-22T02:58:47.346635+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346786+0000","last_change":"2024-09-22T02:58:47.347987+0000","last_active":"2024-09-22T02:58:47.346786+0000","last_peered":"2024-09-22T02:58:47.346786+0000","last_clean":"2024-09-22T02:58:47.346786+0000","last_became_active":"2024-09-22T02:58:45.364066+0000","last_became_peered":"2024-09-22T02:58:45.364066+0000","last_unstale":"2024-09-22T02:58:47.346786+0000","last_undegraded":"2024-09-22T02:58:47.346786+0000","last_fullsized":"2024-09-22T02:58:47.346786+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.344103+0000","last_change":"2024-09-22T02:58:47.346116+0000","last_active":"2024-09-22T02:58:47.344103+0000","last_peered":"2024-09-22T02:58:47.344103+0000","last_clean":"2024-09-22T02:58:47.344103+0000","last_became_active":"2024-09-22T02:58:45.374991+0000","last_became_peered":"2024-09-22T02:58:45.374991+0000","last_unstale":"2024-09-22T02:58:47.344103+0000","last_undegraded":"2024-09-22T02:58:47.344103+0000","last_fullsized":"2024-09-22T02:58:47.344103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.344867+0000","last_change":"2024-09-22T02:58:45.382298+0000","last_active":"2024-09-22T02:58:47.344867+0000","last_peered":"2024-09-22T02:58:47.344867+0000","last_clean":"2024-09-22T02:58:47.344867+0000","last_became_active":"2024-09-22T02:58:45.373807+0000","last_became_peered":"2024-09-22T02:58:45.373807+0000","last_unstale":"2024-09-22T02:58:47.344867+0000","last_undegraded":"2024-09-22T02:58:47.344867+0000","last_fullsized":"2024-09-22T02:58:47.344867+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346501+0000","last_change":"2024-09-22T02:58:47.348407+0000","last_active":"2024-09-22T02:58:47.346501+0000","last_peered":"2024-09-22T02:58:47.346501+0000","last_clean":"2024-09-22T02:58:47.346501+0000","last_became_active":"2024-09-22T02:58:45.369401+0000","last_became_peered":"2024-09-22T02:58:45.369401+0000","last_unstale":"2024-09-22T02:58:47.346501+0000","last_undegraded":"2024-09-22T02:58:47.346501+0000","last_fullsized":"2024-09-22T02:58:47.346501+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346629+0000","last_change":"2024-09-22T02:58:47.346629+0000","last_active":"2024-09-22T02:58:47.346629+0000","last_peered":"2024-09-22T02:58:47.346629+0000","last_clean":"2024-09-22T02:58:47.346629+0000","last_became_active":"2024-09-22T02:58:45.371859+0000","last_became_peered":"2024-09-22T02:58:45.371859+0000","last_unstale":"2024-09-22T02:58:47.346629+0000","last_undegraded":"2024-09-22T02:58:47.346629+0000","last_fullsized":"2024-09-22T02:58:47.346629+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.342743+0000","last_change":"2024-09-22T02:58:47.344427+0000","last_active":"2024-09-22T02:58:47.342743+0000","last_peered":"2024-09-22T02:58:47.342743+0000","last_clean":"2024-09-22T02:58:47.342743+0000","last_became_active":"2024-09-22T02:58:45.366840+0000","last_became_peered":"2024-09-22T02:58:45.366840+0000","last_unstale":"2024-09-22T02:58:47.342743+0000","last_undegraded":"2024-09-22T02:58:47.342743+0000","last_fullsized":"2024-09-22T02:58:47.342743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.343743+0000","last_change":"2024-09-22T02:58:47.345653+0000","last_active":"2024-09-22T02:58:47.343743+0000","last_peered":"2024-09-22T02:58:47.343743+0000","last_clean":"2024-09-22T02:58:47.343743+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T02:58:47.343743+0000","last_undegraded":"2024-09-22T02:58:47.343743+0000","last_fullsized":"2024-09-22T02:58:47.343743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":30757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":180224,"data_stored":621106,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T02:58:52.031 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T02:58:52.031 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-22T02:58:52.031 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-22T02:58:52.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-22T02:58:52.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:52.131+0000 7ff680397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:52.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:52.136+0000 7ff680397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:52.353 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:52.353 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-22T02:58:52.366 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-22T02:58:52.366 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-22T02:58:52.376 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-22T02:58:52.376 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'squid', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-22T02:58:52.376 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-22T02:58:52.376 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-22T02:58:52.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:52.476+0000 7f1757397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:52.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:52.484+0000 7f1757397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:52.678 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-22T02:58:52.678 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T02:58:52.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T02:58:52.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:52.777+0000 7f5261f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:52.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:52.783+0000 7f5261f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:52.960 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:52.961 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T02:58:52.973 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-22T02:58:52.287037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108020,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379340,"statfs":{"total":386547056640,"available":386436444160,"internally_reserved":0,"allocated":671744,"data_stored":1894113,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.842341"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346586+0000","last_change":"2024-09-22T02:58:47.347756+0000","last_active":"2024-09-22T02:58:47.346586+0000","last_peered":"2024-09-22T02:58:47.346586+0000","last_clean":"2024-09-22T02:58:47.346586+0000","last_became_active":"2024-09-22T02:58:45.389306+0000","last_became_peered":"2024-09-22T02:58:45.389306+0000","last_unstale":"2024-09-22T02:58:47.346586+0000","last_undegraded":"2024-09-22T02:58:47.346586+0000","last_fullsized":"2024-09-22T02:58:47.346586+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346635+0000","last_change":"2024-09-22T02:58:47.347783+0000","last_active":"2024-09-22T02:58:47.346635+0000","last_peered":"2024-09-22T02:58:47.346635+0000","last_clean":"2024-09-22T02:58:47.346635+0000","last_became_active":"2024-09-22T02:58:45.383906+0000","last_became_peered":"2024-09-22T02:58:45.383906+0000","last_unstale":"2024-09-22T02:58:47.346635+0000","last_undegraded":"2024-09-22T02:58:47.346635+0000","last_fullsized":"2024-09-22T02:58:47.346635+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346786+0000","last_change":"2024-09-22T02:58:47.347987+0000","last_active":"2024-09-22T02:58:47.346786+0000","last_peered":"2024-09-22T02:58:47.346786+0000","last_clean":"2024-09-22T02:58:47.346786+0000","last_became_active":"2024-09-22T02:58:45.364066+0000","last_became_peered":"2024-09-22T02:58:45.364066+0000","last_unstale":"2024-09-22T02:58:47.346786+0000","last_undegraded":"2024-09-22T02:58:47.346786+0000","last_fullsized":"2024-09-22T02:58:47.346786+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.344103+0000","last_change":"2024-09-22T02:58:47.346116+0000","last_active":"2024-09-22T02:58:47.344103+0000","last_peered":"2024-09-22T02:58:47.344103+0000","last_clean":"2024-09-22T02:58:47.344103+0000","last_became_active":"2024-09-22T02:58:45.374991+0000","last_became_peered":"2024-09-22T02:58:45.374991+0000","last_unstale":"2024-09-22T02:58:47.344103+0000","last_undegraded":"2024-09-22T02:58:47.344103+0000","last_fullsized":"2024-09-22T02:58:47.344103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.344867+0000","last_change":"2024-09-22T02:58:45.382298+0000","last_active":"2024-09-22T02:58:47.344867+0000","last_peered":"2024-09-22T02:58:47.344867+0000","last_clean":"2024-09-22T02:58:47.344867+0000","last_became_active":"2024-09-22T02:58:45.373807+0000","last_became_peered":"2024-09-22T02:58:45.373807+0000","last_unstale":"2024-09-22T02:58:47.344867+0000","last_undegraded":"2024-09-22T02:58:47.344867+0000","last_fullsized":"2024-09-22T02:58:47.344867+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346501+0000","last_change":"2024-09-22T02:58:47.348407+0000","last_active":"2024-09-22T02:58:47.346501+0000","last_peered":"2024-09-22T02:58:47.346501+0000","last_clean":"2024-09-22T02:58:47.346501+0000","last_became_active":"2024-09-22T02:58:45.369401+0000","last_became_peered":"2024-09-22T02:58:45.369401+0000","last_unstale":"2024-09-22T02:58:47.346501+0000","last_undegraded":"2024-09-22T02:58:47.346501+0000","last_fullsized":"2024-09-22T02:58:47.346501+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346629+0000","last_change":"2024-09-22T02:58:47.346629+0000","last_active":"2024-09-22T02:58:47.346629+0000","last_peered":"2024-09-22T02:58:47.346629+0000","last_clean":"2024-09-22T02:58:47.346629+0000","last_became_active":"2024-09-22T02:58:45.371859+0000","last_became_peered":"2024-09-22T02:58:45.371859+0000","last_unstale":"2024-09-22T02:58:47.346629+0000","last_undegraded":"2024-09-22T02:58:47.346629+0000","last_fullsized":"2024-09-22T02:58:47.346629+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.342743+0000","last_change":"2024-09-22T02:58:47.344427+0000","last_active":"2024-09-22T02:58:47.342743+0000","last_peered":"2024-09-22T02:58:47.342743+0000","last_clean":"2024-09-22T02:58:47.342743+0000","last_became_active":"2024-09-22T02:58:45.366840+0000","last_became_peered":"2024-09-22T02:58:45.366840+0000","last_unstale":"2024-09-22T02:58:47.342743+0000","last_undegraded":"2024-09-22T02:58:47.342743+0000","last_fullsized":"2024-09-22T02:58:47.342743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.343743+0000","last_change":"2024-09-22T02:58:47.345653+0000","last_active":"2024-09-22T02:58:47.343743+0000","last_peered":"2024-09-22T02:58:47.343743+0000","last_clean":"2024-09-22T02:58:47.343743+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T02:58:47.343743+0000","last_undegraded":"2024-09-22T02:58:47.343743+0000","last_fullsized":"2024-09-22T02:58:47.343743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":30757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":180224,"data_stored":621106,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T02:58:52.974 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T02:58:53.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:53.073+0000 7fa414d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:53.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:53.080+0000 7fa414d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:53.253 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T02:58:53.265 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-22T02:58:52.287037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108020,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379340,"statfs":{"total":386547056640,"available":386436444160,"internally_reserved":0,"allocated":671744,"data_stored":1894113,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.842341"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346586+0000","last_change":"2024-09-22T02:58:47.347756+0000","last_active":"2024-09-22T02:58:47.346586+0000","last_peered":"2024-09-22T02:58:47.346586+0000","last_clean":"2024-09-22T02:58:47.346586+0000","last_became_active":"2024-09-22T02:58:45.389306+0000","last_became_peered":"2024-09-22T02:58:45.389306+0000","last_unstale":"2024-09-22T02:58:47.346586+0000","last_undegraded":"2024-09-22T02:58:47.346586+0000","last_fullsized":"2024-09-22T02:58:47.346586+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346635+0000","last_change":"2024-09-22T02:58:47.347783+0000","last_active":"2024-09-22T02:58:47.346635+0000","last_peered":"2024-09-22T02:58:47.346635+0000","last_clean":"2024-09-22T02:58:47.346635+0000","last_became_active":"2024-09-22T02:58:45.383906+0000","last_became_peered":"2024-09-22T02:58:45.383906+0000","last_unstale":"2024-09-22T02:58:47.346635+0000","last_undegraded":"2024-09-22T02:58:47.346635+0000","last_fullsized":"2024-09-22T02:58:47.346635+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346786+0000","last_change":"2024-09-22T02:58:47.347987+0000","last_active":"2024-09-22T02:58:47.346786+0000","last_peered":"2024-09-22T02:58:47.346786+0000","last_clean":"2024-09-22T02:58:47.346786+0000","last_became_active":"2024-09-22T02:58:45.364066+0000","last_became_peered":"2024-09-22T02:58:45.364066+0000","last_unstale":"2024-09-22T02:58:47.346786+0000","last_undegraded":"2024-09-22T02:58:47.346786+0000","last_fullsized":"2024-09-22T02:58:47.346786+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.344103+0000","last_change":"2024-09-22T02:58:47.346116+0000","last_active":"2024-09-22T02:58:47.344103+0000","last_peered":"2024-09-22T02:58:47.344103+0000","last_clean":"2024-09-22T02:58:47.344103+0000","last_became_active":"2024-09-22T02:58:45.374991+0000","last_became_peered":"2024-09-22T02:58:45.374991+0000","last_unstale":"2024-09-22T02:58:47.344103+0000","last_undegraded":"2024-09-22T02:58:47.344103+0000","last_fullsized":"2024-09-22T02:58:47.344103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.344867+0000","last_change":"2024-09-22T02:58:45.382298+0000","last_active":"2024-09-22T02:58:47.344867+0000","last_peered":"2024-09-22T02:58:47.344867+0000","last_clean":"2024-09-22T02:58:47.344867+0000","last_became_active":"2024-09-22T02:58:45.373807+0000","last_became_peered":"2024-09-22T02:58:45.373807+0000","last_unstale":"2024-09-22T02:58:47.344867+0000","last_undegraded":"2024-09-22T02:58:47.344867+0000","last_fullsized":"2024-09-22T02:58:47.344867+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346501+0000","last_change":"2024-09-22T02:58:47.348407+0000","last_active":"2024-09-22T02:58:47.346501+0000","last_peered":"2024-09-22T02:58:47.346501+0000","last_clean":"2024-09-22T02:58:47.346501+0000","last_became_active":"2024-09-22T02:58:45.369401+0000","last_became_peered":"2024-09-22T02:58:45.369401+0000","last_unstale":"2024-09-22T02:58:47.346501+0000","last_undegraded":"2024-09-22T02:58:47.346501+0000","last_fullsized":"2024-09-22T02:58:47.346501+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.346629+0000","last_change":"2024-09-22T02:58:47.346629+0000","last_active":"2024-09-22T02:58:47.346629+0000","last_peered":"2024-09-22T02:58:47.346629+0000","last_clean":"2024-09-22T02:58:47.346629+0000","last_became_active":"2024-09-22T02:58:45.371859+0000","last_became_peered":"2024-09-22T02:58:45.371859+0000","last_unstale":"2024-09-22T02:58:47.346629+0000","last_undegraded":"2024-09-22T02:58:47.346629+0000","last_fullsized":"2024-09-22T02:58:47.346629+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.342743+0000","last_change":"2024-09-22T02:58:47.344427+0000","last_active":"2024-09-22T02:58:47.342743+0000","last_peered":"2024-09-22T02:58:47.342743+0000","last_clean":"2024-09-22T02:58:47.342743+0000","last_became_active":"2024-09-22T02:58:45.366840+0000","last_became_peered":"2024-09-22T02:58:45.366840+0000","last_unstale":"2024-09-22T02:58:47.342743+0000","last_undegraded":"2024-09-22T02:58:47.342743+0000","last_fullsized":"2024-09-22T02:58:47.342743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T02:58:47.343743+0000","last_change":"2024-09-22T02:58:47.345653+0000","last_active":"2024-09-22T02:58:47.343743+0000","last_peered":"2024-09-22T02:58:47.343743+0000","last_clean":"2024-09-22T02:58:47.343743+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T02:58:47.343743+0000","last_undegraded":"2024-09-22T02:58:47.343743+0000","last_fullsized":"2024-09-22T02:58:47.343743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":30757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":180224,"data_stored":621106,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T02:58:53.266 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T02:58:53.266 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-22T02:58:53.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:53.368+0000 7f45f8c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:53.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:53.378+0000 7f45f8c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:53.559 INFO:teuthology.orchestra.run.smithi081.stdout:epoch 14 2024-09-22T02:58:53.559 INFO:teuthology.orchestra.run.smithi081.stdout:fsid ba6605a2-3158-4c2e-9364-faeb0f665984 2024-09-22T02:58:53.559 INFO:teuthology.orchestra.run.smithi081.stdout:created 2024-09-22T02:58:34.078383+0000 2024-09-22T02:58:53.559 INFO:teuthology.orchestra.run.smithi081.stdout:modified 2024-09-22T02:58:47.323897+0000 2024-09-22T02:58:53.559 INFO:teuthology.orchestra.run.smithi081.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-22T02:58:53.559 INFO:teuthology.orchestra.run.smithi081.stdout:crush_version 4 2024-09-22T02:58:53.559 INFO:teuthology.orchestra.run.smithi081.stdout:full_ratio 0.95 2024-09-22T02:58:53.559 INFO:teuthology.orchestra.run.smithi081.stdout:backfillfull_ratio 0.9 2024-09-22T02:58:53.559 INFO:teuthology.orchestra.run.smithi081.stdout:nearfull_ratio 0.85 2024-09-22T02:58:53.559 INFO:teuthology.orchestra.run.smithi081.stdout:require_min_compat_client luminous 2024-09-22T02:58:53.559 INFO:teuthology.orchestra.run.smithi081.stdout:min_compat_client jewel 2024-09-22T02:58:53.560 INFO:teuthology.orchestra.run.smithi081.stdout:require_osd_release squid 2024-09-22T02:58:53.560 INFO:teuthology.orchestra.run.smithi081.stdout:stretch_mode_enabled false 2024-09-22T02:58:53.560 INFO:teuthology.orchestra.run.smithi081.stdout:allow_crimson=true 2024-09-22T02:58:53.560 INFO:teuthology.orchestra.run.smithi081.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 14 flags hashpspool,nopgchange,selfmanaged_snaps,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-09-22T02:58:53.560 INFO:teuthology.orchestra.run.smithi081.stdout: removed_snaps_queue [2~1] 2024-09-22T02:58:53.560 INFO:teuthology.orchestra.run.smithi081.stdout:pool 2 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 13 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-09-22T02:58:53.560 INFO:teuthology.orchestra.run.smithi081.stdout:max_osd 4 2024-09-22T02:58:53.560 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.81:6804/3135721300 v2:172.21.15.81:6805/3135721300 exists,up d8dc2c56-1fa8-4c94-b1c4-1ff6bb0919b5 2024-09-22T02:58:53.560 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.81:6802/3974603893 v2:172.21.15.81:6803/3974603893 exists,up 4c95c86d-0d11-4418-921f-c13d799252d2 2024-09-22T02:58:53.560 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6808/294896312 v2:172.21.15.190:6809/294896312 exists,up c70b3bc0-7a70-4652-a939-acac6432f224 2024-09-22T02:58:53.560 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6802/216282597 v2:172.21.15.190:6803/216282597 exists,up 4047b1af-c441-4a8f-bc0d-7c2a4f237e18 2024-09-22T02:58:53.571 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.81:6804/3135721300 v2:172.21.15.81:6805/3135721300 exists,up d8dc2c56-1fa8-4c94-b1c4-1ff6bb0919b5', 'osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.81:6802/3974603893 v2:172.21.15.81:6803/3974603893 exists,up 4c95c86d-0d11-4418-921f-c13d799252d2', 'osd.2 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6808/294896312 v2:172.21.15.190:6809/294896312 exists,up c70b3bc0-7a70-4652-a939-acac6432f224', 'osd.3 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6802/216282597 v2:172.21.15.190:6803/216282597 exists,up 4047b1af-c441-4a8f-bc0d-7c2a4f237e18'] 2024-09-22T02:58:53.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-22T02:58:53.675 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-22T02:58:53.675 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "mon.a", 2024-09-22T02:58:53.675 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster": "ceph", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "admin_socket_mode": "", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_client_required": "cephx, none", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_cluster_required": "cephx", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_debug": "true", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_service_required": "cephx", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_supported": "cephx", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_aio": "true", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-22T02:58:53.676 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_aio_poll_ms": "250", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_aio_reap_max": "16", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_async_discard": "false", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_block_size": "4096", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_debug_aio": "false", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_debug_inflight_ios": "false", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_enable_discard": "false", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_flock_retry": "3", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_inject_crash": "0", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_ioring": "false", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_ioring_hipri": "false", 2024-09-22T02:58:53.677 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bdev_type": "", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_alloc_size": "1048576", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_allocator": "hybrid", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_buffered_io": "true", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_check_for_zeros": "false", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_compact_log_sync": "false", 2024-09-22T02:58:53.678 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_max_log_runway": "4194304", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_max_prefetch": "1048576", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_min_flush_size": "524288", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_min_log_runway": "1048576", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_replay_recovery": "false", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_sync_write": "false", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-22T02:58:53.679 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_allocation_from_file": "true", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_allocator": "hybrid", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bdev_label_multi": "true", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_create": "true", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_db_create": "false", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_db_path": "", 2024-09-22T02:58:53.680 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_db_size": "0", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_path": "", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_preallocate_file": "false", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_size": "107374182400", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_wal_create": "false", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_wal_path": "", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_block_wal_size": "100663296", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bluefs": "true", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-22T02:58:53.681 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_autotune": "true", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_size": "0", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_type": "2q", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-22T02:58:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_clone_cow": "true", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_mode": "none", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_csum_type": "crc32c", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_freelist": "false", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-22T02:58:53.683 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_small_allocations": "0", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_default_buffered_read": "true", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_default_buffered_write": "false", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-22T02:58:53.684 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fail_eio": "false", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-22T02:58:53.685 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_mount": "false", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_umount": "false", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_ignore_data_csum": "false", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-22T02:58:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_log_op_age": "5.000000", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_alloc_size": "0", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_blob_size": "0", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_max_deferred_txc": "32", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_min_alloc_size": "0", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_nid_prealloc": "1024", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-22T02:58:53.687 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_retry_disk_reads": "3", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_rocksdb_cf": "true", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_spdk_mem": "512", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-22T02:58:53.688 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_tracing": "false", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-22T02:58:53.689 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_zero_block_detection": "false", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-22T02:58:53.690 INFO:teuthology.orchestra.run.smithi081.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_cluster_require_signatures": "false", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_cluster_require_version": "2", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_require_signatures": "false", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_require_version": "2", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_service_require_signatures": "false", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_service_require_version": "2", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "cephx_sign_messages": "true", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "chdir": "", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "client_acl_type": "", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "client_asio_thread_count": "2", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "client_cache_mid": "0.750000", 2024-09-22T02:58:53.691 INFO:teuthology.orchestra.run.smithi081.stdout: "client_cache_size": "16384", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_caps_release_delay": "5", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_check_pool_perm": "true", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_debug_force_sync_read": "false", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_debug_getattr_caps": "false", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_debug_inject_features": "", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_debug_inject_tick_delay": "0", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_die_on_failed_remount": "false", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_dirsize_rbytes": "true", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_force_lazyio": "false", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_fs": "", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-22T02:58:53.692 INFO:teuthology.orchestra.run.smithi081.stdout: "client_inject_release_failure": "false", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_max_inline_size": "4096", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_mds_namespace": "", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_metadata": "", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_mount_gid": "-1", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_mount_timeout": "300", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_mount_uid": "-1", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_mountpoint": "/", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_notify_timeout": "10", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc": "true", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc_max_dirty": "104857600", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc_max_objects": "1000", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc_size": "209715200", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_oc_target_dirty": "8388608", 2024-09-22T02:58:53.693 INFO:teuthology.orchestra.run.smithi081.stdout: "client_permissions": "true", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_quota": "true", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_quota_df": "true", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_readahead_max_bytes": "0", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_readahead_max_periods": "4", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_readahead_min": "131072", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_reconnect_stale": "false", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_shutdown_timeout": "30", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_snapdir": ".snap", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_tick_interval": "1", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_trace": "", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_try_dentry_invalidate": "false", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_use_faked_inos": "false", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "client_use_random_mds": "false", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_graylog": "false", 2024-09-22T02:58:53.694 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_graylog_port": "12201", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_monitors": "default=true", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_syslog": "false", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "clog_to_syslog_level": "info", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_addr": "-", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_network": "", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_network_interface": "", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "compressor_zlib_isal": "false", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "compressor_zlib_level": "5", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "compressor_zlib_winsize": "-15", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "compressor_zstd_level": "1", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-22T02:58:53.695 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_alien_op_num_threads": "6", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_osd_stat_interval": "0", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_seastar_cpu_cores": "", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "crimson_seastar_num_threads": "0", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_location": "", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_location_hook": "", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "crush_location_hook_timeout": "10", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "daemonize": "false", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "daos_pool": "tank", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-22T02:58:53.696 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_alienstore": "0/5", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_allow_any_pool_priority": "false", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_asok": "1/5", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_asok_assert_abort": "false", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_asserts_on_shutdown": "true", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_auth": "20/20", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_bdev": "1/3", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_bluefs": "1/5", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_bluestore": "1/5", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_buffer": "0/1", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_ceph_exporter": "1/5", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_cephfs_mirror": "0/5", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_cephsqlite": "0/5", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_client": "0/5", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_compressor": "1/5", 2024-09-22T02:58:53.697 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_context": "0/1", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_crimson_interrupt": "0/5", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_crush": "1/1", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_crypto": "1/5", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_cyanstore": "0/5", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_deliberately_leak_memory": "false", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_disable_randomized_ping": "false", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_dpdk": "1/5", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_eventtrace": "1/5", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_filer": "0/1", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_filestore": "1/3", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_finisher": "1/1", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_fuse": "1/5", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_heartbeat_testing_span": "0", 2024-09-22T02:58:53.698 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_heartbeatmap": "1/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_javaclient": "1/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_journal": "1/3", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_journaler": "0/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_kstore": "1/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_lockdep": "0/1", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mclock": "1/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds": "1/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_balancer": "1/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_locker": "1/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_log": "1/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_log_expire": "1/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_migrator": "1/5", 2024-09-22T02:58:53.699 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mds_quiesce": "3/5", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_memstore": "1/5", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mgr": "2/5", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mgrc": "1/5", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_mon": "20/20", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_monc": "0/10", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_ms": "1/1", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_none": "0/5", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_objclass": "0/5", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_objectcacher": "0/5", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_objecter": "0/1", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_optracker": "0/5", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_osd": "1/5", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_paxos": "20/20", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_perfcounter": "1/5", 2024-09-22T02:58:53.700 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_prioritycache": "1/5", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rados": "0/5", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rbd": "0/5", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rbd_mirror": "0/5", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rbd_pwl": "0/5", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rbd_replay": "0/5", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_refs": "0/0", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_reserver": "1/1", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw": "1/5", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_access": "1/5", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_datacache": "1/5", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_dbstore": "1/5", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_flight": "1/5", 2024-09-22T02:58:53.701 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rgw_sync": "1/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_rocksdb": "4/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore": "0/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_backref": "0/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_cache": "0/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_cleaner": "0/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_device": "0/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_epm": "0/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_journal": "0/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_lba": "0/5", 2024-09-22T02:58:53.702 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_odata": "0/5", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_omap": "0/5", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_onode": "0/5", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_t": "0/5", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_seastore_tm": "0/5", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_striper": "0/1", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_test": "0/5", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_throttle": "1/1", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_timer": "0/1", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_tp": "0/5", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "debug_trace": "1/5", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "device_failure_prediction_mode": "none", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-09-22T02:58:53.703 INFO:teuthology.orchestra.run.smithi081.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "err_to_graylog": "false", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "err_to_journald": "false", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "err_to_stderr": "true", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "err_to_syslog": "false", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "event_tracing": "false", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_addr": "0.0.0.0", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_http_port": "9926", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_prio_limit": "5", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_sort_metrics": "true", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "exporter_stats_period": "5", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "fake_statfs_for_testing": "0", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "fatal_signal_handlers": "true", 2024-09-22T02:58:53.704 INFO:teuthology.orchestra.run.smithi081.stdout: "filer_max_purge_ops": "10", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filer_max_truncate_ops": "128", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_apply_finisher_threads": "1", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_blackhole": "false", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_btrfs_clone_range": "true", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_btrfs_snap": "true", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_caller_concurrency": "10", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_collect_device_partition_information": "true", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_commit_timeout": "600.000000", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_debug_inject_read_err": "false", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_debug_omap_check": "false", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_debug_verify_split": "false", 2024-09-22T02:58:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_dump_file": "", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fadvise": "true", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fail_eio": "true", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fd_cache_shards": "16", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fd_cache_size": "128", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fiemap": "false", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fiemap_threshold": "4096", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_inject_stall": "0", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_journal_parallel": "false", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_journal_trailing": "false", 2024-09-22T02:58:53.706 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_journal_writeahead": "false", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_kill_at": "0", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattrs": "0", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_xattr_value_size": "0", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-22T02:58:53.707 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_merge_threshold": "-10", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_odsync_write": "false", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_omap_backend": "rocksdb", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_omap_backend_path": "", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_op_thread_timeout": "60", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_op_threads": "2", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_punch_hole": "false", 2024-09-22T02:58:53.708 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_queue_max_ops": "50", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_seek_data_hole": "false", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_sloppy_crc": "false", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-22T02:58:53.709 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_splice": "false", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_split_multiple": "2", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_split_rand_factor": "20", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_update_to": "1000", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_enable": "true", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-22T02:58:53.710 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_xfs_extsize": "false", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "filestore_zfs_snap": "false", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "fio_dir": "/tmp/fio", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "fsid": "ba6605a2-3158-4c2e-9364-faeb0f665984", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_allow_other": "true", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_atomic_o_trunc": "true", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_big_writes": "true", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_debug": "false", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_default_permissions": "false", 2024-09-22T02:58:53.711 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_disable_pagecache": "false", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_max_write": "0", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_multithreaded": "true", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_require_active_mds": "true", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_set_user_groups": "true", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_splice_move": "true", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_splice_read": "true", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_splice_write": "true", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "fuse_use_invalidate_cb": "true", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "gss_target_name": "ceph", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_file": "", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_inject_failure": "0", 2024-09-22T02:58:53.712 INFO:teuthology.orchestra.run.smithi081.stdout: "heartbeat_interval": "5", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "host": "smithi081", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-22T02:58:53.713 INFO:teuthology.orchestra.run.smithi081.stdout: "inject_early_sigterm": "false", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "jaeger_agent_port": "6799", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "jaeger_tracing_enable": "false", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_aio": "true", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_align_min_size": "65536", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_block_align": "true", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_block_size": "4096", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_dio": "true", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_discard": "false", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_force_aio": "false", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_ignore_corruption": "false", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_max_write_bytes": "10485760", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_max_write_entries": "100", 2024-09-22T02:58:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_replay_from": "0", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_write_header_frequency": "0", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "journal_zero_on_create": "false", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "journaler_prefetch_periods": "10", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "journaler_prezero_periods": "5", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "journaler_write_head_interval": "15", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "keyfile": "", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_backend": "rocksdb", 2024-09-22T02:58:53.715 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_default_stripe_size": "65536", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_fsck_on_mount": "false", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_max_bytes": "67108864", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_max_ops": "512", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_nid_prealloc": "1024", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_onode_map_size": "1024", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_sync_submit_transaction": "false", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "kstore_sync_transaction": "false", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "librados_thread_count": "2", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "lockdep": "false", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "lockdep_force_backtrace": "false", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "log_coarse_timestamps": "true", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-22T02:58:53.716 INFO:teuthology.orchestra.run.smithi081.stdout: "log_flush_on_exit": "false", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "log_graylog_host": "127.0.0.1", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "log_graylog_port": "12201", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "log_max_new": "1000", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "log_max_recent": "10000", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "log_stderr_prefix": "", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "log_stop_at_utilization": "0.970000", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "log_to_file": "true", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "log_to_graylog": "false", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "log_to_journald": "false", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "log_to_stderr": "false", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "log_to_syslog": "false", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "max_rotating_auth_attempts": "10", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-22T02:58:53.717 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_action_on_write_error": "1", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_alternate_name_max": "8192", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_asio_thread_count": "2", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_export_pin": "true", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_fragment_dirs": "true", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_fragment_interval": "5", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_interval": "10", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_max": "-1", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_max_until": "-1", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_merge_size": "50", 2024-09-22T02:58:53.718 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_midchunk": "0.300000", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_min_start": "0.200000", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_minchunk": "0.001000", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_mode": "0", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_need_max": "1.200000", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_need_min": "0.800000", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_overload_epochs": "2", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_split_bits": "3", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_split_size": "10000", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-22T02:58:53.719 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_target_decay": "10.000000", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_beacon_grace": "15.000000", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_beacon_interval": "4.000000", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_mid": "0.700000", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_quiesce_delay": "0", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_release_free_interval": "10", 2024-09-22T02:58:53.720 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_reservation": "0.050000", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_trim_interval": "1", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache_trim_threshold": "262144", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_client_prealloc_inos": "1000", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_connect_bootstrapping": "false", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_damage_table_max_entries": "10000", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_debug_auth_pins": "false", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_debug_frag": "false", 2024-09-22T02:58:53.721 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_debug_scatterstat": "false", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_debug_subtrees": "false", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_decay_halflife": "5.000000", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_default_dir_hash": "2", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_defer_session_stale": "true", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_deny_all_reconnect": "false", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dir_keys_per_op": "16384", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dir_max_commit_size": "10", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dir_max_entries": "0", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dir_prefetch": "true", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dump_cache_on_map": "false", 2024-09-22T02:58:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_early_reply": "true", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_enable_op_tracker": "true", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_enforce_unique_name": "true", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_export_ephemeral_random": "true", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-22T02:58:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_health_summarize_threshold": "10", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_health_dummy": "false", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_migrator_session_race": "false", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_join_fs": "", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_journal_format": "1", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-22T02:58:53.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_create_at": "0", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_export_at": "0", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_import_at": "0", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_journal_at": "0", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_journal_expire_at": "0", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_journal_replay_at": "0", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_link_at": "0", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_mdstable_at": "0", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_openc_at": "0", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_rename_at": "0", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_kill_shutdown_at": "0", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_event_large_threshold": "524288", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_events_per_segment": "1024", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-22T02:58:53.725 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_max_events": "-1", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_max_segments": "128", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_pause": "false", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_segment_size": "0", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_trim_threshold": "128", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log_warn_factor": "2.000000", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_caps_per_client": "1048576", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_completed_flushes": "100000", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_completed_requests": "100000", 2024-09-22T02:58:53.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_export_size": "20971520", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_file_recover": "32", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_purge_files": "64", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_purge_ops": "8192", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_max_snaps_per_dir": "100", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_metrics_update_interval": "2", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_min_caps_per_client": "100", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_min_caps_working_set": "10000", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_numa_node": "-1", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_complaint_time": "30.000000", 2024-09-22T02:58:53.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_history_duration": "600", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_history_size": "20", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_history_slow_op_size": "20", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_op_log_threshold": "5", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_ping_grace": "15", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_ping_interval": "5", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_max_caps": "30000", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-22T02:58:53.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_recall_warning_threshold": "262144", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_replay_interval": "1.000000", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_root_ino_gid": "0", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_root_ino_uid": "0", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-22T02:58:53.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_shutdown_check": "0", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_skip_ino": "0", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_snap_max_uid": "4294967294", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_snap_min_uid": "0", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_snap_rstat": "false", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_standby_replay_damaged": "false", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_symlink_recovery": "true", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_thrash_exports": "0", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_thrash_fragments": "0", 2024-09-22T02:58:53.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_tick_interval": "5.000000", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_valgrind_exit": "false", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_verify_backtrace": "1", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_verify_scatter": "false", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_wipe_sessions": "false", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mempool_debug": "false", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "memstore_device_bytes": "1073741824", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "memstore_page_set": "false", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "memstore_page_size": "65536", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_client_bytes": "134217728", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_client_messages": "512", 2024-09-22T02:58:53.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_disabled_modules": "", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_enable_op_tracker": "true", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_max_pg_num_change": "128", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_mds_bytes": "134217728", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_mds_messages": "128", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_mon_bytes": "134217728", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_mon_messages": "128", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-22T02:58:53.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_history_duration": "600", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_history_size": "20", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_op_log_threshold": "5", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_osd_bytes": "536870912", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_osd_messages": "8192", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_pool": "true", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_standby_modules": "true", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_stats_period": "5", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_stats_threshold": "5", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_tick_period": "2", 2024-09-22T02:58:53.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_allow_pool_delete": "true", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_allow_pool_size_one": "true", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_auth_validate_all_caps": "true", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_bytes": "104857600", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_directed_command_retry": "2", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_hunt_parallel": "3", 2024-09-22T02:58:53.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_log_interval": "1.000000", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_ping_interval": "10.000000", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_client_target_rank": "-1", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_file_level": "debug", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_file": "true", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-22T02:58:53.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_journald": "false", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_compact_on_bootstrap": "false", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_compact_on_start": "false", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_compact_on_trim": "true", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_cpu_threads": "4", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_crush_min_required_version": "hammer", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_daemon_bytes": "419430400", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-22T02:58:53.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_data_avail_crit": "5", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_data_avail_warn": "5", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_data_size_warn": "16106127360", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_dump_json": "false", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_dump_transactions": "false", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_extra_checks": "false", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_no_require_reef": "false", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_no_require_squid": "false", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-22T02:58:53.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_down_mkfs_grace": "120", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_down_uptime_grace": "60", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_election_default_strategy": "1", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_election_timeout": "5.000000", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_elector_ping_divisor": "2", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_enable_op_tracker": "true", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_fake_pool_delete": "false", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_force_quorum_join": "false", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_globalid_prealloc": "10000", 2024-09-22T02:58:53.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_detail_to_clog": "false", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_log_update_period": "5", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_max_detail": "50", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_to_clog": "true", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_to_clog_interval": "600", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_host": "172.21.15.81", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_host_override": "", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_initial_members": "", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_lease": "5.000000", 2024-09-22T02:58:53.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_log_full_interval": "50", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_log_max": "10000", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_log_max_summary": "50", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_log_epochs": "500", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_osd": "10000", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_pg_per_osd": "10000", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_pool_pg_num": "65536", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-22T02:58:53.740 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mds_force_trim_to": "0", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mds_skip_sanity": "false", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_memory_autotune": "true", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_memory_target": "2147483648", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_beacon_grace": "30", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_digest_period": "5", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_inactive_grace": "60", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_min_osdmap_epochs": "50", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_complaint_time": "30", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_history_duration": "600", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_history_size": "20", 2024-09-22T02:58:53.741 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_history_slow_op_size": "20", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_op_log_threshold": "5", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_auto_mark_in": "false", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_cache_size": "500", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-22T02:58:53.742 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_down_out_interval": "600", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_force_trim_to": "0", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-22T02:58:53.743 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_min_down_reporters": "2", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_report_timeout": "900", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-22T02:58:53.744 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_stuck_threshold": "60", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_probe_timeout": "2.000000", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_reweight_max_change": "0.050000", 2024-09-22T02:58:53.745 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_reweight_max_osds": "4", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_scrub_interval": "86400", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_scrub_max_keys": "100", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_scrub_timeout": "300", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_session_timeout": "300", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_smart_report_timeout": "5", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_stat_smooth_intervals": "6", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-22T02:58:53.746 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_stretch_pool_min_size": "2", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_stretch_pool_size": "4", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_debug": "false", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_provider_kill_at": "0", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_requester_kill_at": "0", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_sync_timeout": "60.000000", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_target_pg_per_osd": "100", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_tick_interval": "5", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_timecheck_interval": "300.000000", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_use_min_delay_socket": "false", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_older_version_delay": "604800", 2024-09-22T02:58:53.747 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_misplaced": "false", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_older_version": "true", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-22T02:58:53.748 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "monmap": "", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "motr_tracing_enabled": "false", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_op_threads": "3", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_cm": "false", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_device_name": "", 2024-09-22T02:58:53.749 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_dscp": "96", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_local_gid": "", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_port_num": "1", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_sl": "3", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_support_srq": "true", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_rdma_type": "ib", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_async_reap_threshold": "5", 2024-09-22T02:58:53.750 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_before_connect": "false", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_ipv4": "true", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_ipv6": "false", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_msgr1": "true", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_msgr2": "true", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_port_max": "7568", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_port_min": "6800", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_retry_count": "3", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_bind_retry_delay": "5", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_blackhole_client": "false", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_blackhole_mds": "false", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_blackhole_mgr": "false", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_blackhole_mon": "false", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_blackhole_osd": "false", 2024-09-22T02:58:53.751 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_client_mode": "crc", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_cluster_mode": "crc", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_cluster_type": "", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_compress_secure": "false", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_connection_idle_timeout": "900", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_connection_ready_timeout": "10", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_crc_data": "true", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_crc_header": "true", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_die_on_bad_msg": "false", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_die_on_bug": "true", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_die_on_old_message": "true", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_die_on_skipped_message": "false", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-22T02:58:53.752 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_coremask": "0xF", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_enable_tso": "true", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_hugepages": "", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_lro": "true", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_memory_channel": "4", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_pmd": "", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_port_id": "0", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-22T02:58:53.753 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_dump_on_send": "false", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_initial_backoff": "0.200000", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_delay_max": "1.000000", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_delay_type": "", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_network_congestion": "0", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_inject_socket_failures": "0", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_learn_addr_from_peer": "true", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_max_accept_failures": "4", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_max_backoff": "15.000000", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_mon_client_mode": "crc", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_mon_cluster_mode": "crc", 2024-09-22T02:58:53.754 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_mon_service_mode": "crc", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_osd_compress_min_size": "1024", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_osd_compress_mode": "none", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_pq_min_cost": "65536", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_public_type": "", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_service_mode": "crc", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_tcp_listen_backlog": "512", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_tcp_nodelay": "true", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_tcp_rcvbuf": "0", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "ms_type": "async+posix", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "no_config_file": "false", 2024-09-22T02:58:53.755 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_completion_locks_per_session": "32", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_inflight_ops": "1024", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_tick_interval": "5.000000", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter_timeout": "10.000000", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "objectstore_blackhole": "false", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "openssl_engine_opts": "", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_delay_time": "5.000000", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_hist_halflife": "1000", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_max_low_ops": "2", 2024-09-22T02:58:53.756 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_max_ops": "4", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_agent_slop": "0.020000", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_asio_thread_count": "2", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_async_recovery_min_cost": "100", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backfill_scan_max": "512", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backfill_scan_min": "64", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backoff_on_degraded": "false", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backoff_on_peering": "false", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_backoff_on_unfound": "true", 2024-09-22T02:58:53.757 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_beacon_report_interval": "300", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bench_duration": "30", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bench_max_block_size": "67108864", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_blkin_trace_all": "false", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_check_for_log_corruption": "false", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-22T02:58:53.758 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_class_update_on_start": "true", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_client_message_cap": "256", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_client_message_size_cap": "524288000", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_client_op_priority": "63", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_client_watch_timeout": "30", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_command_max_records": "256", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_compact_on_start": "false", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_crush_update_on_start": "true", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_crush_update_weight_set": "true", 2024-09-22T02:58:53.759 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_feed_pullee": "-1", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_misdirected_ops": "false", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_no_acting_change": "false", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_no_purge_strays": "false", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_op_order": "false", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-22T02:58:53.760 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_shutdown": "false", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_trim_objects": "false", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_verify_snaps": "false", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_keys": "1024", 2024-09-22T02:58:53.761 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_stride": "524288", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_default_notify_timeout": "30", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_delete_sleep": "0.000000", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_discard_disconnected_ops": "true", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_enable_op_tracker": "true", 2024-09-22T02:58:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_fast_info": "true", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_fast_shutdown": "true", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_function_tracing": "false", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_grace": "20", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_interval": "6", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-22T02:58:53.763 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_min_peers": "10", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_min_size": "2000", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_stale": "600", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_hit_set_max_size": "100000", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_hit_set_min_size": "1000", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_journal_size": "5120", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_kill_backfill_at": "0", 2024-09-22T02:58:53.764 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_map_cache_size": "50", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_map_dedup": "true", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_map_message_max": "40", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_map_share_max_epochs": "40", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_attr_name_len": "100", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_attr_size": "0", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_backfills": "1", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_markdown_count": "5", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_markdown_period": "600", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_object_name_len": "2048", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_object_namespace_len": "256", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_object_size": "134217728", 2024-09-22T02:58:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_pg_blocked_by": "16", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_pg_log_entries": "10000", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_pgls": "1024", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_push_cost": "8388608", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_push_objects": "10", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_scrubs": "3", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_trimming_pgs": "2", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_write_op_reply_len": "64", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_max_write_size": "90", 2024-09-22T02:58:53.766 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_profile": "balanced", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-22T02:58:53.767 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_base": "805306368", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_cache_min": "134217728", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_target": "4294967296", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_target_autotune": "false", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_min_pg_log_entries": "250", 2024-09-22T02:58:53.768 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_min_recovery_priority": "0", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mon_report_interval": "5", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_num_cache_shards": "32", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_num_op_tracker_shard": "32", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_numa_auto_affinity": "true", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_numa_node": "-1", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_numa_prefer_iface": "true", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_objecter_finishers": "1", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_objectstore": "bluestore", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_objectstore_fuse": "false", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_objectstore_tracing": "false", 2024-09-22T02:58:53.769 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_complaint_time": "30.000000", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_history_duration": "600", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_history_size": "20", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_history_slow_op_size": "20", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_log_threshold": "5", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_shards": "0", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_shards_hdd": "5", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_shards_ssd": "8", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_pq_min_cost": "65536", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-22T02:58:53.770 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_queue_cut_off": "high", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_op_thread_timeout": "15", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_open_classes_on_start": "true", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_os_flags": "0", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_peering_op_priority": "255", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_delete_cost": "1048576", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_delete_priority": "5", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_log_trim_max": "10000", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_log_trim_min": "100", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-22T02:58:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_crimson": "true", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-22T02:58:53.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_flags": "0", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_min_size": "2", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_pg_num": "32", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_pgp_num": "0", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_read_ratio": "70", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_size": "3", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_default_type": "replicated", 2024-09-22T02:58:53.773 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_push_per_object_cost": "1000", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recover_clone_overlap": "true", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_cost": "20971520", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_active": "0", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_max_single_start": "1", 2024-09-22T02:58:53.774 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_op_priority": "3", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_priority": "5", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_sleep": "0.000000", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_repair_during_recovery": "false", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_requested_scrub_priority": "120", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_auto_repair": "false", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-22T02:58:53.775 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_begin_hour": "0", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_begin_week_day": "0", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_chunk_max": "25", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_chunk_min": "5", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_cost": "52428800", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_during_recovery": "false", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_end_hour": "0", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_end_week_day": "0", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_event_cost": "4096", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_invalid_stats": "true", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-22T02:58:53.776 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_max_preemptions": "5", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_priority": "5", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_scrub_sleep": "0.000000", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_skip_data_digest": "false", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_smart_report_timeout": "5", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_cost": "1048576", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_priority": "5", 2024-09-22T02:58:53.777 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_target_transaction_size": "30", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-22T02:58:53.778 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_tracing": "false", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_use_stale_snap": "false", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "osdc_blkin_trace_all": "false", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_kill_at": "0", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_max_join_drift": "10", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_min": "500", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_min_wait": "0.050000", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_propose_interval": "1.000000", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_service_trim_max": "500", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-22T02:58:53.779 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_service_trim_min": "10", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_stash_full_interval": "25", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_trim_max": "500", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "paxos_trim_min": "250", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "perf": "true", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addr": "-", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "public_addrv": "", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "public_bind_addr": "-", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "public_network": "", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "public_network_interface": "", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "qat_compressor_busy_polling": "false", 2024-09-22T02:58:53.780 INFO:teuthology.orchestra.run.smithi081.stdout: "qat_compressor_enabled": "false", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "qat_compressor_session_max_number": "256", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rados_mon_op_timeout": "0", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rados_osd_op_timeout": "0", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rados_tracing": "false", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_atime_update_interval": "60", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_balance_parent_reads": "false", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_balance_snap_reads": "false", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_blkin_trace_all": "false", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache": "true", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-22T02:58:53.781 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_policy": "writearound", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_size": "33554432", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_clone_copy_on_read": "false", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_compression_hint": "none", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_concurrent_management_ops": "10", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_clone_format": "auto", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_data_pool": "", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_features": "61", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_format": "2", 2024-09-22T02:58:53.782 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_map_options": "", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_order": "22", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_pool": "rbd", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_stripe_count": "0", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_default_stripe_unit": "0", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_enable_alloc_hint": "true", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_io_scheduler": "simple", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-22T02:58:53.783 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_order": "24", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_pool": "", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_journal_splay_width": "4", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_localize_parent_reads": "false", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_localize_snap_reads": "false", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-22T02:58:53.784 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-22T02:58:53.785 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-22T02:58:53.786 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_mtime_update_interval": "60", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_non_blocking_aio": "true", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_op_thread_timeout": "60", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_op_threads": "1", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_parent_cache_enabled": "false", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_plugins": "", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_bps_burst": "0", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_bps_limit": "0", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_exclude_ops": "0", 2024-09-22T02:58:53.787 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_iops_burst": "0", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_iops_limit": "0", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-22T02:58:53.788 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_read_from_replica_policy": "default", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_skip_partial_discard": "true", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_tracing": "false", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_validate_names": "true", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rbd_validate_pool": "true", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "restapi_base_url": "", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "restapi_log_level": "", 2024-09-22T02:58:53.789 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_acl_grants_max_num": "100", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_admin_entry": "admin", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_backend_store": "rados", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_barbican_url": "", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_beast_enable_async": "true", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_counters_cache": "false", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-22T02:58:53.790 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_cache_enabled": "true", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_cache_expiry_interval": "900", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_cache_lru_size": "25000", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_config_store": "rados", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_content_length_compat": "false", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_copy_obj_progress": "true", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_cors_rules_max_num": "100", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_cross_domain_policy": "", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-22T02:58:53.791 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_addr": "", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_password": "", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_kmip_username": "", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_require_ssl": "true", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-22T02:58:53.792 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_addr": "", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_auth": "token", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_namespace": "", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_prefix": "", 2024-09-22T02:58:53.793 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_token_file": "", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_curl_buffersize": "524288", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_curl_low_speed_time": "300", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-22T02:58:53.794 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_d4n_port": "6379", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_log_changes_size": "1000", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_log_num_shards": "128", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_log_window": "30", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-22T02:58:53.795 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-22T02:58:53.796 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dns_name": "", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dns_s3website_name": "", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_dynamic_resharding": "true", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_gc_threads": "true", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_lc_threads": "true", 2024-09-22T02:58:53.797 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_ops_log": "false", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_quota_threads": "true", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_static_website": "false", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enable_usage_log": "false", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_enforce_swift_acls": "true", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_exit_timeout_secs": "120", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_expose_bucket": "false", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_extended_http_attrs": "", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_filter": "none", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_frontends": "beast port=7480", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_deferred": "50", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-22T02:58:53.798 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_objs": "32", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_gc_processor_period": "3600", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_init_timeout": "300", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_kafka_connection_idle": "300", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-22T02:58:53.799 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_domain": "", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_password": "", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_password_path": "", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_project": "", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_tenant": "", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_token": "", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_token_path": "", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_admin_user": "", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_api_version": "2", 2024-09-22T02:58:53.800 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_barbican_domain": "", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_barbican_password": "", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_barbican_project": "", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_barbican_user": "", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_url": "", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_debug_interval": "-1", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_lock_max_time": "90", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_max_objs": "32", 2024-09-22T02:58:53.801 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_max_rules": "1000", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_max_worker": "3", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lc_thread_delay": "0", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_dnattr": "uid", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_searchfilter": "", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ldap_uri": "", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_log_http_headers": "", 2024-09-22T02:58:53.802 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_log_object_name_utc": "false", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_attr_name_len": "0", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_attr_size": "0", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_chunk_size": "4194304", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_listing_results": "1000", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_notify_retries": "10", 2024-09-22T02:58:53.803 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_put_param_size": "1048576", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_put_size": "5368709120", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_max_slo_entries": "1000", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_md_log_max_shards": "64", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_mp_lock_max_time": "600", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-22T02:58:53.804 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_max_gc": "300", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_num_async_rados_threads": "32", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_num_control_oids": "8", 2024-09-22T02:58:53.805 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_numa_node": "-1", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_objexp_chunk_size": "100", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_objexp_gc_interval": "600", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_op_thread_timeout": "600", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_op_tracing": "false", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_opa_token": "", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_opa_url": "", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_opa_verify_ssl": "true", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-22T02:58:53.806 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ops_log_rados": "false", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_ops_log_socket_path": "", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_period_push_interval": "2.000000", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_cache_lanes": "3", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_cache_partitions": "3", 2024-09-22T02:58:53.807 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_print_continue": "true", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_rados_tracing": "false", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_realm": "", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_realm_id": "", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_region": "", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-22T02:58:53.808 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_request_uri": "", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_batch_size": "64", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_max_aio": "128", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_num_logs": "16", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_reshard_thread_interval": "600", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_resolve_cname": "false", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_run_sync_thread": "true", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-22T02:58:53.809 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_scheduler_type": "throttler", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_script_uri": "", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_service_provider_name": "", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_client_id": "", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_client_secret": "", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_entry": "sts", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_key": "sts", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-22T02:58:53.810 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_min_session_duration": "900", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sts_token_introspection_url": "", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_account_in_url": "false", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_auth_entry": "auth", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_auth_url": "", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_custom_header": "", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_need_stats": "true", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_tenant_name": "", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_token_expiration": "86400", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_url": "", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_url_prefix": "swift", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-22T02:58:53.811 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_lease_period": "120", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_thread_pool_size": "512", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-22T02:58:53.812 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_comment": "", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_createby": "", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_encoding": "", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_flag": "false", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_origin": "", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_torrent_tracker": "", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_trust_forwarded_https": "false", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_usage_max_shards": "32", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_usage_max_user_shards": "1", 2024-09-22T02:58:53.813 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_use_opa_authz": "false", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_counters_cache": "false", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_max_buckets": "1000", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_policies_max_num": "100", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_user_unique_email": "true", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_verify_ssl": "true", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-22T02:58:53.814 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zone": "", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zone_id": "", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zonegroup": "", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zonegroup_id": "", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rgwlc_auto_session_clear": "true", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_block_size": "4096", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_size": "536870912", 2024-09-22T02:58:53.815 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_index_type": "binary_search", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_partition_filters": "false", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_perf": "false", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-22T02:58:53.816 INFO:teuthology.orchestra.run.smithi081.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "run_dir": "/var/run/ceph", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_block_create": "true", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_cache_lru_size": "67108864", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_default_max_object_size": "16777216", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_device_size": "53687091200", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_full_integrity_check": "false", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_journal_batch_capacity": "16", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-22T02:58:53.817 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_main_device_type": "SSD", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "seastore_segment_size": "67108864", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "set_keepcaps": "false", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "setgroup": "", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "setuser": "", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "setuser_match_path": "", 2024-09-22T02:58:53.818 INFO:teuthology.orchestra.run.smithi081.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-22T02:58:53.819 INFO:teuthology.orchestra.run.smithi081.stdout: "thp": "false", 2024-09-22T02:58:53.819 INFO:teuthology.orchestra.run.smithi081.stdout: "threadpool_default_timeout": "60", 2024-09-22T02:58:53.819 INFO:teuthology.orchestra.run.smithi081.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-22T02:58:53.819 INFO:teuthology.orchestra.run.smithi081.stdout: "throttler_perf_counter": "true" 2024-09-22T02:58:53.819 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-22T02:58:53.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-22T02:58:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:53.919+0000 7f2ec1597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:53.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:53.924+0000 7f2ec1597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:53.938 INFO:teuthology.orchestra.run.smithi081.stdout:{} 2024-09-22T02:58:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-22T02:58:53.948 DEBUG:teuthology.orchestra.run.smithi081:> type ceph-objectstore-tool 2024-09-22T02:58:53.963 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-22T02:58:53.963 DEBUG:teuthology.orchestra.run.smithi190:> type ceph-objectstore-tool 2024-09-22T02:58:53.981 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-22T02:58:53.982 INFO:teuthology.run_tasks:Running task radosbench... 2024-09-22T02:58:54.033 INFO:tasks.radosbench:Beginning radosbench... 2024-09-22T02:58:54.033 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-22T02:58:54.033 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-09-22T02:58:54.034 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-22T02:58:54.034 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T02:58:54.034 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T02:58:54.034 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T02:58:54.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T02:58:54.035 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-22T02:58:54.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:58:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.139+0000 7f7488997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.143+0000 7f7488997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.139+0000 7effd6397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.139+0000 7f3084397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.147+0000 7effd5135640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.150+0000 7f3084397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.169 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:58:54.169 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:58:54.169 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:58:54.169 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:58:54.169 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:58:54.169 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:58:54.169 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:58:54.169 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:58:54.169 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:58:54.169 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:58:54.169 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:58:54.170 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:58:54.173 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:58:54.173 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:58:54.173 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T02:58:54.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.277+0000 7fd87b6d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.285+0000 7fd87b6d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.299 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:58:54.299 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:58:54.299 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:58:54.299 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:58:54.299 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:58:54.299 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:58:54.299 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:58:54.299 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:58:54.299 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:58:54.299 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:58:54.300 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:58:54.300 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:58:54.301 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:58:54.302 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:58:54.302 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T02:58:54.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.402+0000 7f9323197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.409+0000 7f9323197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.424 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:58:54.424 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:58:54.424 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:58:54.425 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:58:54.425 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:58:54.425 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:58:54.425 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:58:54.425 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:58:54.425 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:58:54.425 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:58:54.425 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:58:54.425 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:58:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:58:54.427 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:58:54.428 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T02:58:54.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.526+0000 7f6eaf397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.534+0000 7f6eaf397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:58:54.549 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:58:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:58:54.552 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:58:54.667 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_0' created 2024-09-22T02:58:54.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-22T02:58:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.794+0000 7f588e957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:54.806+0000 7f588e957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:55.668 INFO:teuthology.orchestra.run.smithi081.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-22T02:58:56.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:58:56.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:56.328+0000 7fadda8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:56.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:56.332+0000 7fadda8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:56.692 DEBUG:teuthology.orchestra.run.smithi081:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 150 write' 2024-09-22T02:58:56.695 INFO:teuthology.run_tasks:Running task admin_socket... 2024-09-22T02:58:58.084 DEBUG:tasks.admin_socket:Running admin socket tests on client.0 2024-09-22T02:58:58.084 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -- /home/ubuntu/cephtest/admin_socket_client.0 && while test '!' -e /var/run/ceph/ceph-client.0.asok ; do sleep 1 ; done 2024-09-22T02:58:58.086 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:hints = 1 2024-09-22T02:58:58.086 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 150 seconds or 0 objects 2024-09-22T02:58:58.086 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object prefix: benchmark_data_smithi081_30037 2024-09-22T02:58:58.086 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T02:58:58.086 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 0 0 0 0 0 0 - 0 2024-09-22T02:58:58.086 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 1 16 476 460 28.7483 28.75 0.0339257 0.034172 2024-09-22T02:58:58.108 DEBUG:tasks.admin_socket:Testing objecter_requests with config {'test': 'http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}', 'branch': 'squid'} 2024-09-22T02:58:58.108 DEBUG:teuthology.orchestra.run.smithi081:> wget -q -O /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests -- 'http://git.ceph.com/?p=ceph.git;a=blob_plain;f=src/test/admin_socket/objecter_requests;hb=squid' && chmod u=rx -- /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests 2024-09-22T02:58:58.401 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:58:58.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:58.560+0000 7f61a146c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:58.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:58.568+0000 7f61a146c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:58.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-client.0.asok objecter_requests 2024-09-22T02:58:58.751 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 2 16 943 927 28.9666 29.1875 0.0474257 0.0340906 2024-09-22T02:58:58.855 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-22T02:58:58.855 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-09-22T02:58:58.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 975, 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.5b0dd63a", 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object974", 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object974", 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.528888s", 2024-09-22T02:58:58.856 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.034999310999999998, 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.819546+0000", 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 983, 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.ac943faa", 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-22T02:58:58.857 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object982", 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object982", 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.545887s", 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.017999646000000001, 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.836729+0000", 2024-09-22T02:58:58.858 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 991, 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.550bc2ef", 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 0, 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object990", 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object990", 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.859 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.562887s", 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.00099997999999999992, 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.853376+0000", 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 972, 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.777ccadd", 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object971", 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object971", 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.519888s", 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.043999134000000002, 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.810809+0000", 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 977, 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.350f52d2", 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object976", 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object976", 2024-09-22T02:58:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.533887s", 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.029999409000000001, 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.824607+0000", 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 981, 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.bae263f5", 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object980", 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object980", 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.544887s", 2024-09-22T02:58:58.864 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.018999625999999999, 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.834939+0000", 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 984, 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.a337a8fd", 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-22T02:58:58.865 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object983", 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object983", 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.545887s", 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.017999646000000001, 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.836859+0000", 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.866 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 985, 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.c5e88228", 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object984", 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object984", 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.867 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.547887s", 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.015999685, 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.838810+0000", 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 987, 2024-09-22T02:58:58.868 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.a94f478", 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 1, 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object986", 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object986", 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.555887s", 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0079998420000000001, 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.869 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.846345+0000", 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 982, 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.900c2054", 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object981", 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object981", 2024-09-22T02:58:58.870 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.544887s", 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.018999625999999999, 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.835777+0000", 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 986, 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.c4cd24f6", 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 2, 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object985", 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object985", 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.550887s", 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.012999744000000001, 2024-09-22T02:58:58.872 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.841303+0000", 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 979, 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.8c3c3157", 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object978", 2024-09-22T02:58:58.873 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object978", 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.538887s", 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.024999508, 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.829616+0000", 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.874 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 980, 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.fcb89f21", 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object979", 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object979", 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.543887s", 2024-09-22T02:58:58.875 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.019999606, 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.834210+0000", 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 988, 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.f3ecf137", 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-22T02:58:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object987", 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object987", 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.557887s", 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0059998819999999998, 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.848156+0000", 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.877 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 989, 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.1558b51b", 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object988", 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object988", 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.878 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.560887s", 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.0029999409999999999, 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.851385+0000", 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 990, 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.c5e64e11", 2024-09-22T02:58:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object989", 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object989", 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.562887s", 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.00099997999999999992, 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.853029+0000", 2024-09-22T02:58:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 992, 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: "pg": "3.1ca371e0", 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": 3, 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: "object_id": "benchmark_data_smithi081_30037_object991", 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: "object_locator": "@3", 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_id": "benchmark_data_smithi081_30037_object991", 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: "target_object_locator": "@3", 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: "paused": 0, 2024-09-22T02:58:58.881 INFO:teuthology.orchestra.run.smithi081.stdout: "used_replica": 0, 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: "precalc_pgid": 0, 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: "last_sent": "506.563887s", 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0, 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: "attempts": 1, 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: "snapid": "head", 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: "snap_context": "0=[]", 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: "mtime": "2024-09-22T02:58:58.854037+0000", 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_ops": [ 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: "write 0~65536 in=65536b" 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: ], 2024-09-22T02:58:58.882 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_ops": [], 2024-09-22T02:58:58.883 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_ops": [], 2024-09-22T02:58:58.883 INFO:teuthology.orchestra.run.smithi081.stdout: "pool_stat_ops": [], 2024-09-22T02:58:58.883 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_ops": [], 2024-09-22T02:58:58.883 INFO:teuthology.orchestra.run.smithi081.stdout: "command_ops": [] 2024-09-22T02:58:58.883 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-22T02:58:58.884 DEBUG:tasks.admin_socket:admin socket command objecter_requests returned { "ops": [ { "tid": 975, "pg": "3.5b0dd63a", "osd": 0, "object_id": "benchmark_data_smithi081_30037_object974", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object974", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.528888s", "age": 0.034999310999999998, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.819546+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 983, "pg": "3.ac943faa", "osd": 0, "object_id": "benchmark_data_smithi081_30037_object982", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object982", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.545887s", "age": 0.017999646000000001, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.836729+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 991, "pg": "3.550bc2ef", "osd": 0, "object_id": "benchmark_data_smithi081_30037_object990", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object990", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.562887s", "age": 0.00099997999999999992, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.853376+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 972, "pg": "3.777ccadd", "osd": 1, "object_id": "benchmark_data_smithi081_30037_object971", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object971", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.519888s", "age": 0.043999134000000002, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.810809+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 977, "pg": "3.350f52d2", "osd": 1, "object_id": "benchmark_data_smithi081_30037_object976", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object976", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.533887s", "age": 0.029999409000000001, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.824607+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 981, "pg": "3.bae263f5", "osd": 1, "object_id": "benchmark_data_smithi081_30037_object980", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object980", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.544887s", "age": 0.018999625999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.834939+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 984, "pg": "3.a337a8fd", "osd": 1, "object_id": "benchmark_data_smithi081_30037_object983", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object983", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.545887s", "age": 0.017999646000000001, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.836859+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 985, "pg": "3.c5e88228", "osd": 1, "object_id": "benchmark_data_smithi081_30037_object984", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object984", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.547887s", "age": 0.015999685, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.838810+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 987, "pg": "3.a94f478", "osd": 1, "object_id": "benchmark_data_smithi081_30037_object986", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object986", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.555887s", "age": 0.0079998420000000001, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.846345+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 982, "pg": "3.900c2054", "osd": 2, "object_id": "benchmark_data_smithi081_30037_object981", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object981", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.544887s", "age": 0.018999625999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.835777+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 986, "pg": "3.c4cd24f6", "osd": 2, "object_id": "benchmark_data_smithi081_30037_object985", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object985", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.550887s", "age": 0.012999744000000001, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.841303+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 979, "pg": "3.8c3c3157", "osd": 3, "object_id": "benchmark_data_smithi081_30037_object978", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object978", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.538887s", "age": 0.024999508, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.829616+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 980, "pg": "3.fcb89f21", "osd": 3, "object_id": "benchmark_data_smithi081_30037_object979", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object979", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.543887s", "age": 0.019999606, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.834210+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 988, "pg": "3.f3ecf137", "osd": 3, "object_id": "benchmark_data_smithi081_30037_object987", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object987", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.557887s", "age": 0.0059998819999999998, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.848156+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 989, "pg": "3.1558b51b", "osd": 3, "object_id": "benchmark_data_smithi081_30037_object988", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object988", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.560887s", "age": 0.0029999409999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.851385+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 990, "pg": "3.c5e64e11", "osd": 3, "object_id": "benchmark_data_smithi081_30037_object989", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object989", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.562887s", "age": 0.00099997999999999992, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.853029+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 992, "pg": "3.1ca371e0", "osd": 3, "object_id": "benchmark_data_smithi081_30037_object991", "object_locator": "@3", "target_object_id": "benchmark_data_smithi081_30037_object991", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "506.563887s", "age": 0, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-22T02:58:58.854037+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] } ], "linger_ops": [], "pool_ops": [], "pool_stat_ops": [], "statfs_ops": [], "command_ops": [] } 2024-09-22T02:58:58.884 DEBUG:teuthology.orchestra.run.smithi081:> /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests 2024-09-22T02:58:58.941 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/admin_socket_client.0 2024-09-22T02:58:58.964 DEBUG:teuthology.parallel:result is None 2024-09-22T02:58:58.964 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2024-09-22T02:58:58.981 INFO:tasks.radosbench:joining radosbench (timing out after 4800s) 2024-09-22T02:58:58.981 INFO:teuthology.orchestra.run:waiting for 4800 2024-09-22T02:58:59.553 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T02:58:59.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T02:58:59.553 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T02:58:59.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:59.706+0000 7f38dce63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:59.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:58:59.716+0000 7f38dce63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:58:59.751 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 3 16 1411 1395 29.0603 29.25 0.0468441 0.0342211 2024-09-22T02:58:59.997 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:58:59.997 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T02:59:00.019 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-22T02:58:58.288116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13304371,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":333,"num_write_kb":14312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":282,"ondisk_log_size":282,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108796,"kb_used_data":1468,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377378564,"statfs":{"total":386547056640,"available":386435649536,"internally_reserved":0,"allocated":1503232,"data_stored":14178277,"data_compressed":671551,"data_compressed_allocated":888832,"data_compressed_original":13975552,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12713984,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":194,"num_write_kb":12416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001078"},"pg_stats":[{"pgid":"3.f","version":"17'8","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.210324+0000","last_change":"2024-09-22T02:58:55.762293+0000","last_active":"2024-09-22T02:58:57.210324+0000","last_peered":"2024-09-22T02:58:57.210324+0000","last_clean":"2024-09-22T02:58:57.210324+0000","last_became_active":"2024-09-22T02:58:55.757845+0000","last_became_peered":"2024-09-22T02:58:55.757845+0000","last_unstale":"2024-09-22T02:58:57.210324+0000","last_undegraded":"2024-09-22T02:58:57.210324+0000","last_fullsized":"2024-09-22T02:58:57.210324+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'17","reported_seq":30,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.261143+0000","last_change":"2024-09-22T02:58:55.755437+0000","last_active":"2024-09-22T02:58:57.261143+0000","last_peered":"2024-09-22T02:58:57.261143+0000","last_clean":"2024-09-22T02:58:57.261143+0000","last_became_active":"2024-09-22T02:58:55.750996+0000","last_became_peered":"2024-09-22T02:58:55.750996+0000","last_unstale":"2024-09-22T02:58:57.261143+0000","last_undegraded":"2024-09-22T02:58:57.261143+0000","last_fullsized":"2024-09-22T02:58:57.261143+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1179648,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-22T02:58:55.787582+0000","last_change":"2024-09-22T02:58:55.788334+0000","last_active":"2024-09-22T02:58:55.787582+0000","last_peered":"2024-09-22T02:58:55.787582+0000","last_clean":"2024-09-22T02:58:54.663941+0000","last_became_active":"2024-09-22T02:58:55.779349+0000","last_became_peered":"2024-09-22T02:58:55.779349+0000","last_unstale":"2024-09-22T02:58:55.787582+0000","last_undegraded":"2024-09-22T02:58:55.787582+0000","last_fullsized":"2024-09-22T02:58:55.787582+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'14","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.271529+0000","last_change":"2024-09-22T02:58:55.740009+0000","last_active":"2024-09-22T02:58:57.271529+0000","last_peered":"2024-09-22T02:58:57.271529+0000","last_clean":"2024-09-22T02:58:57.271529+0000","last_became_active":"2024-09-22T02:58:55.731779+0000","last_became_peered":"2024-09-22T02:58:55.731779+0000","last_unstale":"2024-09-22T02:58:57.271529+0000","last_undegraded":"2024-09-22T02:58:57.271529+0000","last_fullsized":"2024-09-22T02:58:57.271529+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'18","reported_seq":31,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.234544+0000","last_change":"2024-09-22T02:58:55.742415+0000","last_active":"2024-09-22T02:58:57.234544+0000","last_peered":"2024-09-22T02:58:57.234544+0000","last_clean":"2024-09-22T02:58:57.234544+0000","last_became_active":"2024-09-22T02:58:55.732221+0000","last_became_peered":"2024-09-22T02:58:55.732221+0000","last_unstale":"2024-09-22T02:58:57.234544+0000","last_undegraded":"2024-09-22T02:58:57.234544+0000","last_fullsized":"2024-09-22T02:58:57.234544+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1245184,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'13","reported_seq":26,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.176791+0000","last_change":"2024-09-22T02:58:55.768855+0000","last_active":"2024-09-22T02:58:57.176791+0000","last_peered":"2024-09-22T02:58:57.176791+0000","last_clean":"2024-09-22T02:58:57.176791+0000","last_became_active":"2024-09-22T02:58:55.764760+0000","last_became_peered":"2024-09-22T02:58:55.764760+0000","last_unstale":"2024-09-22T02:58:57.176791+0000","last_undegraded":"2024-09-22T02:58:57.176791+0000","last_fullsized":"2024-09-22T02:58:57.176791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'14","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.166026+0000","last_change":"2024-09-22T02:58:55.757520+0000","last_active":"2024-09-22T02:58:57.166026+0000","last_peered":"2024-09-22T02:58:57.166026+0000","last_clean":"2024-09-22T02:58:57.166026+0000","last_became_active":"2024-09-22T02:58:55.752439+0000","last_became_peered":"2024-09-22T02:58:55.752439+0000","last_unstale":"2024-09-22T02:58:57.166026+0000","last_undegraded":"2024-09-22T02:58:57.166026+0000","last_fullsized":"2024-09-22T02:58:57.166026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-22T02:58:55.753993+0000","last_change":"2024-09-22T02:58:55.754878+0000","last_active":"2024-09-22T02:58:55.753993+0000","last_peered":"2024-09-22T02:58:55.753993+0000","last_clean":"2024-09-22T02:58:54.663941+0000","last_became_active":"2024-09-22T02:58:55.750260+0000","last_became_peered":"2024-09-22T02:58:55.750260+0000","last_unstale":"2024-09-22T02:58:55.753993+0000","last_undegraded":"2024-09-22T02:58:55.753993+0000","last_fullsized":"2024-09-22T02:58:55.753993+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":25,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.689703+0000","last_change":"2024-09-22T02:58:47.345653+0000","last_active":"2024-09-22T02:58:56.689703+0000","last_peered":"2024-09-22T02:58:56.689703+0000","last_clean":"2024-09-22T02:58:56.689703+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T02:58:56.689703+0000","last_undegraded":"2024-09-22T02:58:56.689703+0000","last_fullsized":"2024-09-22T02:58:56.689703+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-22T02:58:55.765660+0000","last_change":"2024-09-22T02:58:55.766536+0000","last_active":"2024-09-22T02:58:55.765660+0000","last_peered":"2024-09-22T02:58:55.765660+0000","last_clean":"2024-09-22T02:58:54.663941+0000","last_became_active":"2024-09-22T02:58:55.761477+0000","last_became_peered":"2024-09-22T02:58:55.761477+0000","last_unstale":"2024-09-22T02:58:55.765660+0000","last_undegraded":"2024-09-22T02:58:55.765660+0000","last_fullsized":"2024-09-22T02:58:55.765660+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":25,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.689190+0000","last_change":"2024-09-22T02:58:47.344427+0000","last_active":"2024-09-22T02:58:56.689190+0000","last_peered":"2024-09-22T02:58:56.689190+0000","last_clean":"2024-09-22T02:58:56.689190+0000","last_became_active":"2024-09-22T02:58:45.366840+0000","last_became_peered":"2024-09-22T02:58:45.366840+0000","last_unstale":"2024-09-22T02:58:56.689190+0000","last_undegraded":"2024-09-22T02:58:56.689190+0000","last_fullsized":"2024-09-22T02:58:56.689190+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'16","reported_seq":29,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.269522+0000","last_change":"2024-09-22T02:58:55.741407+0000","last_active":"2024-09-22T02:58:57.269522+0000","last_peered":"2024-09-22T02:58:57.269522+0000","last_clean":"2024-09-22T02:58:57.269522+0000","last_became_active":"2024-09-22T02:58:55.737022+0000","last_became_peered":"2024-09-22T02:58:55.737022+0000","last_unstale":"2024-09-22T02:58:57.269522+0000","last_undegraded":"2024-09-22T02:58:57.269522+0000","last_fullsized":"2024-09-22T02:58:57.269522+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.689097+0000","last_change":"2024-09-22T02:58:47.346629+0000","last_active":"2024-09-22T02:58:56.689097+0000","last_peered":"2024-09-22T02:58:56.689097+0000","last_clean":"2024-09-22T02:58:56.689097+0000","last_became_active":"2024-09-22T02:58:45.371859+0000","last_became_peered":"2024-09-22T02:58:45.371859+0000","last_unstale":"2024-09-22T02:58:56.689097+0000","last_undegraded":"2024-09-22T02:58:56.689097+0000","last_fullsized":"2024-09-22T02:58:56.689097+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"17'14","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.297193+0000","last_change":"2024-09-22T02:58:55.752486+0000","last_active":"2024-09-22T02:58:57.297193+0000","last_peered":"2024-09-22T02:58:57.297193+0000","last_clean":"2024-09-22T02:58:57.297193+0000","last_became_active":"2024-09-22T02:58:55.746225+0000","last_became_peered":"2024-09-22T02:58:55.746225+0000","last_unstale":"2024-09-22T02:58:57.297193+0000","last_undegraded":"2024-09-22T02:58:57.297193+0000","last_fullsized":"2024-09-22T02:58:57.297193+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":26,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.692727+0000","last_change":"2024-09-22T02:58:47.348407+0000","last_active":"2024-09-22T02:58:56.692727+0000","last_peered":"2024-09-22T02:58:56.692727+0000","last_clean":"2024-09-22T02:58:56.692727+0000","last_became_active":"2024-09-22T02:58:45.369401+0000","last_became_peered":"2024-09-22T02:58:45.369401+0000","last_unstale":"2024-09-22T02:58:56.692727+0000","last_undegraded":"2024-09-22T02:58:56.692727+0000","last_fullsized":"2024-09-22T02:58:56.692727+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":151,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.691481+0000","last_change":"2024-09-22T02:58:45.382298+0000","last_active":"2024-09-22T02:58:56.691481+0000","last_peered":"2024-09-22T02:58:56.691481+0000","last_clean":"2024-09-22T02:58:56.691481+0000","last_became_active":"2024-09-22T02:58:45.373807+0000","last_became_peered":"2024-09-22T02:58:45.373807+0000","last_unstale":"2024-09-22T02:58:56.691481+0000","last_undegraded":"2024-09-22T02:58:56.691481+0000","last_fullsized":"2024-09-22T02:58:56.691481+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'15","reported_seq":28,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.275274+0000","last_change":"2024-09-22T02:58:55.773825+0000","last_active":"2024-09-22T02:58:57.275274+0000","last_peered":"2024-09-22T02:58:57.275274+0000","last_clean":"2024-09-22T02:58:57.275274+0000","last_became_active":"2024-09-22T02:58:55.767555+0000","last_became_peered":"2024-09-22T02:58:55.767555+0000","last_unstale":"2024-09-22T02:58:57.275274+0000","last_undegraded":"2024-09-22T02:58:57.275274+0000","last_fullsized":"2024-09-22T02:58:57.275274+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"17'20","reported_seq":33,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.265226+0000","last_change":"2024-09-22T02:58:55.747999+0000","last_active":"2024-09-22T02:58:57.265226+0000","last_peered":"2024-09-22T02:58:57.265226+0000","last_clean":"2024-09-22T02:58:57.265226+0000","last_became_active":"2024-09-22T02:58:55.743521+0000","last_became_peered":"2024-09-22T02:58:55.743521+0000","last_unstale":"2024-09-22T02:58:57.265226+0000","last_undegraded":"2024-09-22T02:58:57.265226+0000","last_fullsized":"2024-09-22T02:58:57.265226+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-22T02:58:55.691778+0000","last_change":"2024-09-22T02:58:47.346116+0000","last_active":"2024-09-22T02:58:55.691778+0000","last_peered":"2024-09-22T02:58:55.691778+0000","last_clean":"2024-09-22T02:58:55.691778+0000","last_became_active":"2024-09-22T02:58:45.374991+0000","last_became_peered":"2024-09-22T02:58:45.374991+0000","last_unstale":"2024-09-22T02:58:55.691778+0000","last_undegraded":"2024-09-22T02:58:55.691778+0000","last_fullsized":"2024-09-22T02:58:55.691778+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"17'18","reported_seq":31,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.253232+0000","last_change":"2024-09-22T02:58:55.759688+0000","last_active":"2024-09-22T02:58:57.253232+0000","last_peered":"2024-09-22T02:58:57.253232+0000","last_clean":"2024-09-22T02:58:57.253232+0000","last_became_active":"2024-09-22T02:58:55.754829+0000","last_became_peered":"2024-09-22T02:58:55.754829+0000","last_unstale":"2024-09-22T02:58:57.253232+0000","last_undegraded":"2024-09-22T02:58:57.253232+0000","last_fullsized":"2024-09-22T02:58:57.253232+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1245184,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":25,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.692870+0000","last_change":"2024-09-22T02:58:47.347987+0000","last_active":"2024-09-22T02:58:56.692870+0000","last_peered":"2024-09-22T02:58:56.692870+0000","last_clean":"2024-09-22T02:58:56.692870+0000","last_became_active":"2024-09-22T02:58:45.364066+0000","last_became_peered":"2024-09-22T02:58:45.364066+0000","last_unstale":"2024-09-22T02:58:56.692870+0000","last_undegraded":"2024-09-22T02:58:56.692870+0000","last_fullsized":"2024-09-22T02:58:56.692870+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'15","reported_seq":28,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:57.283543+0000","last_change":"2024-09-22T02:58:55.740822+0000","last_active":"2024-09-22T02:58:57.283543+0000","last_peered":"2024-09-22T02:58:57.283543+0000","last_clean":"2024-09-22T02:58:57.283543+0000","last_became_active":"2024-09-22T02:58:55.733222+0000","last_became_peered":"2024-09-22T02:58:55.733222+0000","last_unstale":"2024-09-22T02:58:57.283543+0000","last_undegraded":"2024-09-22T02:58:57.283543+0000","last_fullsized":"2024-09-22T02:58:57.283543+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-22T02:58:55.696745+0000","last_change":"2024-09-22T02:58:47.347783+0000","last_active":"2024-09-22T02:58:55.696745+0000","last_peered":"2024-09-22T02:58:55.696745+0000","last_clean":"2024-09-22T02:58:55.696745+0000","last_became_active":"2024-09-22T02:58:45.383906+0000","last_became_peered":"2024-09-22T02:58:45.383906+0000","last_unstale":"2024-09-22T02:58:55.696745+0000","last_undegraded":"2024-09-22T02:58:55.696745+0000","last_fullsized":"2024-09-22T02:58:55.696745+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-22T02:58:55.781327+0000","last_change":"2024-09-22T02:58:55.782093+0000","last_active":"2024-09-22T02:58:55.781327+0000","last_peered":"2024-09-22T02:58:55.781327+0000","last_clean":"2024-09-22T02:58:54.663941+0000","last_became_active":"2024-09-22T02:58:55.771463+0000","last_became_peered":"2024-09-22T02:58:55.771463+0000","last_unstale":"2024-09-22T02:58:55.781327+0000","last_undegraded":"2024-09-22T02:58:55.781327+0000","last_fullsized":"2024-09-22T02:58:55.781327+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-22T02:58:55.693277+0000","last_change":"2024-09-22T02:58:47.347756+0000","last_active":"2024-09-22T02:58:55.693277+0000","last_peered":"2024-09-22T02:58:55.693277+0000","last_clean":"2024-09-22T02:58:55.693277+0000","last_became_active":"2024-09-22T02:58:45.389306+0000","last_became_peered":"2024-09-22T02:58:45.389306+0000","last_unstale":"2024-09-22T02:58:55.693277+0000","last_undegraded":"2024-09-22T02:58:55.693277+0000","last_fullsized":"2024-09-22T02:58:55.693277+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":12713984,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":194,"num_write_kb":12416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2416640,"data_stored":38666240,"data_compressed":1835247,"data_compressed_allocated":2416640,"data_compressed_original":38666240,"omap_allocated":0,"internal_metadata":0},"log_size":182,"ondisk_log_size":182,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26968,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344872,"statfs":{"total":96636764160,"available":96609148928,"internally_reserved":0,"allocated":139264,"data_stored":37998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27784,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":974848,"data_stored":12887203,"data_compressed":611633,"data_compressed_allocated":806912,"data_compressed_original":12828672,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":196608,"data_stored":628347,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705678,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":192512,"data_stored":624729,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":696320,"data_stored":11141120,"data_compressed":528803,"data_compressed_allocated":696320,"data_compressed_original":11141120,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":774144,"data_stored":12386304,"data_compressed":587896,"data_compressed_allocated":774144,"data_compressed_original":12386304,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":946176,"data_stored":15138816,"data_compressed":718548,"data_compressed_allocated":946176,"data_compressed_original":15138816,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T02:59:00.751 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 4 16 1888 1872 29.2478 29.8125 0.0478509 0.0340735 2024-09-22T02:59:00.754 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:00.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:00.908+0000 7fc0e842c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:00.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:00.916+0000 7fc0e842c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:01.751 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 5 16 2327 2311 28.8853 27.4375 0.0353383 0.0344986 2024-09-22T02:59:02.751 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 6 16 2791 2775 28.9041 29 0.0231823 0.0344735 2024-09-22T02:59:03.195 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:03.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:03.342+0000 7f7063997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:03.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:03.349+0000 7f7063997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:03.751 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 7 16 3238 3222 28.7657 27.9375 0.0359319 0.0346583 2024-09-22T02:59:04.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 8 16 3716 3700 28.9041 29.875 0.0537429 0.0345171 2024-09-22T02:59:05.021 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T02:59:05.021 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T02:59:05.021 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T02:59:05.022 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T02:59:05.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:05.161+0000 7fddd5a4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:05.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:05.173+0000 7fddd5a4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:05.213 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:05.213 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:05.217 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:05.217 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T02:59:05.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:05.356+0000 7f226966c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:05.373+0000 7f226966c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:05.444 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:05.444 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:05.444 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:05.444 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:05.444 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:05.444 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:05.444 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:05.445 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:05.445 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:05.445 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:05.445 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:05.445 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:05.448 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:05.449 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:05.449 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T02:59:05.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:05.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:05.609+0000 7f4e4210e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:05.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:05.628+0000 7f4e4210e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:05.652 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:05.652 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:05.652 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:05.656 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:05.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T02:59:05.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:05.708+0000 7f4b55317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:05.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:05.721+0000 7f4b55317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:05.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 9 16 4145 4129 28.6714 26.8125 0.0329117 0.0348025 2024-09-22T02:59:05.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:05.792+0000 7f342b557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:05.805+0000 7f342b557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:05.852 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:05.852 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:05.852 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:05.852 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:05.853 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:05.853 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:05.853 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:05.853 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:05.853 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:05.853 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:05.853 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:05.853 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:05.858 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:05.858 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:06.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 10 16 4598 4582 28.6353 28.3125 0.02782 0.0348554 2024-09-22T02:59:07.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 11 16 5052 5036 28.6114 28.375 0.0213002 0.034874 2024-09-22T02:59:07.978 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:08.119+0000 7fd145757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:08.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:08.127+0000 7fd145757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:08.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 12 16 5497 5481 28.5446 27.8125 0.0181369 0.0349667 2024-09-22T02:59:09.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 13 16 5935 5919 28.4545 27.375 0.025243 0.0350323 2024-09-22T02:59:10.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:10.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:10.478+0000 7ff3bd86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:10.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:10.485+0000 7ff3bd86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:10.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 14 16 6365 6349 28.3415 26.875 0.0316081 0.0352326 2024-09-22T02:59:10.859 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T02:59:10.859 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T02:59:10.859 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T02:59:10.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T02:59:11.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:11.009+0000 7fd0facd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:11.022+0000 7fd0facd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:11.056 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:11.056 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:11.056 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:11.057 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:11.057 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:11.057 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:11.057 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:11.057 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:11.057 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:11.057 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:11.057 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:11.057 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:11.060 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:11.060 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T02:59:11.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:11.210+0000 7ff56d197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:11.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:11.224+0000 7ff56d197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:11.300 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:11.300 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:11.300 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:11.300 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:11.300 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:11.300 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:11.300 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:11.300 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:11.300 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:11.300 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:11.301 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:11.301 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:11.303 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:11.304 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:11.304 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T02:59:11.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:11.454+0000 7fabb674e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:11.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:11.467+0000 7fabb674e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:11.489 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:11.489 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:11.489 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:11.489 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:11.489 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:11.489 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:11.489 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:11.489 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:11.489 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:11.489 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:11.489 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:11.490 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:11.496 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:11.496 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T02:59:11.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:11.642+0000 7f4b726d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:11.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:11.656+0000 7f4b726d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:11.694 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:11.697 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:11.698 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:11.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 15 16 6788 6772 28.2144 26.4375 0.0262017 0.0353973 2024-09-22T02:59:12.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:12.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 16 16 7216 7200 28.1228 26.75 0.0387466 0.035501 2024-09-22T02:59:12.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:12.854+0000 7fc49d82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:12.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:12.862+0000 7fc49d82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:13.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 17 16 7620 7604 27.9537 25.25 0.044735 0.0357261 2024-09-22T02:59:14.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 18 16 8047 8031 27.8833 26.6875 0.0232515 0.0358332 2024-09-22T02:59:15.134 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:15.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:15.282+0000 7f39ab157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:15.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:15.289+0000 7f39ab157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:15.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 19 16 8464 8448 27.7873 26.0625 0.0274001 0.0359383 2024-09-22T02:59:16.698 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T02:59:16.698 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T02:59:16.699 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T02:59:16.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T02:59:16.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-22T02:59:16.752247+0000 min lat: 0.0084511 max lat: 0.112472 avg lat: 0.0360272 2024-09-22T02:59:16.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T02:59:16.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 20 16 8891 8875 27.7322 26.6875 0.0315437 0.0360272 2024-09-22T02:59:16.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:16.842+0000 7fabd7197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:16.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:16.856+0000 7fabd7197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:16.918 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:16.918 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:16.919 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:16.919 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:16.919 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:16.919 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:16.919 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:16.919 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:16.919 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:16.919 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:16.919 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:16.919 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:16.922 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:16.922 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T02:59:17.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:17.084+0000 7fced0d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:17.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:17.099+0000 7fced0d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:17.137 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:17.137 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:17.137 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:17.137 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:17.137 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:17.137 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:17.137 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:17.141 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:17.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T02:59:17.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:17.290+0000 7f048b863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:17.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:17.305+0000 7f048b863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:17.330 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:17.330 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:17.330 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:17.330 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:17.331 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:17.331 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:17.331 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:17.331 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:17.331 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:17.331 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:17.331 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:17.331 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:17.335 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:17.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T02:59:17.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:17.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:17.475+0000 7fa4f2f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:17.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:17.492+0000 7fa4f2f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:17.537 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:17.538 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:17.538 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:17.538 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:17.538 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:17.538 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:17.538 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:17.538 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:17.539 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:17.539 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:17.539 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:17.539 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:17.543 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:17.544 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:17.632+0000 7f2627d4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:17.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:17.644+0000 7f2627d4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:17.752 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 21 16 9302 9286 27.6347 25.6875 0.0552496 0.0361395 2024-09-22T02:59:18.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 22 16 9733 9717 27.6029 26.9375 0.0350593 0.0361924 2024-09-22T02:59:19.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 23 16 10134 10118 27.4924 25.0625 0.0255029 0.0363181 2024-09-22T02:59:19.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:20.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:20.030+0000 7f21cf797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:20.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:20.038+0000 7f21cf797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:20.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 24 16 10535 10519 27.391 25.0625 0.0308841 0.0364633 2024-09-22T02:59:21.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 25 16 10956 10940 27.3478 26.3125 0.0443165 0.0365353 2024-09-22T02:59:22.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:22.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:22.420+0000 7f8356797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:22.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:22.426+0000 7f8356797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:22.544 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T02:59:22.544 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T02:59:22.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T02:59:22.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:22.698+0000 7fd4bf597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:22.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:22.707+0000 7fd4bf597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:22.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 26 16 11352 11336 27.2478 24.75 0.0201246 0.0366584 2024-09-22T02:59:23.015 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:59:23.015 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T02:59:23.036 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-22T02:59:22.292277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":626197043,"num_objects":9550,"num_object_clones":0,"num_object_copies":28650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":9685,"num_write_kb":612840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9630,"ondisk_log_size":9630,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":246532,"kb_used_data":110080,"kb_used_omap":0,"kb_used_meta":136384,"kb_avail":377240828,"statfs":{"total":386547056640,"available":386294607872,"internally_reserved":0,"allocated":112721920,"data_stored":1793167449,"data_compressed":85143731,"data_compressed_allocated":112074752,"data_compressed_original":1792950272,"omap_allocated":0,"internal_metadata":139657216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":314834944,"num_objects":4804,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4804,"num_write_kb":307456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002119"},"pg_stats":[{"pgid":"3.f","version":"17'562","reported_seq":575,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.198841+0000","last_change":"2024-09-22T02:58:55.762293+0000","last_active":"2024-09-22T02:59:17.198841+0000","last_peered":"2024-09-22T02:59:17.198841+0000","last_clean":"2024-09-22T02:59:17.198841+0000","last_became_active":"2024-09-22T02:58:55.757845+0000","last_became_peered":"2024-09-22T02:58:55.757845+0000","last_unstale":"2024-09-22T02:59:17.198841+0000","last_undegraded":"2024-09-22T02:59:17.198841+0000","last_fullsized":"2024-09-22T02:59:17.198841+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36896768,"num_objects":563,"num_object_clones":0,"num_object_copies":1689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":563,"num_write_kb":36032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'555","reported_seq":568,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.164543+0000","last_change":"2024-09-22T02:58:55.755437+0000","last_active":"2024-09-22T02:59:17.164543+0000","last_peered":"2024-09-22T02:59:17.164543+0000","last_clean":"2024-09-22T02:59:17.164543+0000","last_became_active":"2024-09-22T02:58:55.750996+0000","last_became_peered":"2024-09-22T02:58:55.750996+0000","last_unstale":"2024-09-22T02:59:17.164543+0000","last_undegraded":"2024-09-22T02:59:17.164543+0000","last_fullsized":"2024-09-22T02:59:17.164543+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36438016,"num_objects":556,"num_object_clones":0,"num_object_copies":1668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":556,"num_write_kb":35584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'682","reported_seq":695,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:21.232705+0000","last_change":"2024-09-22T02:58:55.788334+0000","last_active":"2024-09-22T02:59:21.232705+0000","last_peered":"2024-09-22T02:59:21.232705+0000","last_clean":"2024-09-22T02:59:21.232705+0000","last_became_active":"2024-09-22T02:58:55.779349+0000","last_became_peered":"2024-09-22T02:58:55.779349+0000","last_unstale":"2024-09-22T02:59:21.232705+0000","last_undegraded":"2024-09-22T02:59:21.232705+0000","last_fullsized":"2024-09-22T02:59:21.232705+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":683,"num_object_clones":0,"num_object_copies":2049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":683,"num_write_kb":43712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'552","reported_seq":565,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.177821+0000","last_change":"2024-09-22T02:58:55.740009+0000","last_active":"2024-09-22T02:59:17.177821+0000","last_peered":"2024-09-22T02:59:17.177821+0000","last_clean":"2024-09-22T02:59:17.177821+0000","last_became_active":"2024-09-22T02:58:55.731779+0000","last_became_peered":"2024-09-22T02:58:55.731779+0000","last_unstale":"2024-09-22T02:59:17.177821+0000","last_undegraded":"2024-09-22T02:59:17.177821+0000","last_fullsized":"2024-09-22T02:59:17.177821+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36241408,"num_objects":553,"num_object_clones":0,"num_object_copies":1659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":553,"num_write_kb":35392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'549","reported_seq":562,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.289827+0000","last_change":"2024-09-22T02:58:55.742415+0000","last_active":"2024-09-22T02:59:17.289827+0000","last_peered":"2024-09-22T02:59:17.289827+0000","last_clean":"2024-09-22T02:59:17.289827+0000","last_became_active":"2024-09-22T02:58:55.732221+0000","last_became_peered":"2024-09-22T02:58:55.732221+0000","last_unstale":"2024-09-22T02:59:17.289827+0000","last_undegraded":"2024-09-22T02:59:17.289827+0000","last_fullsized":"2024-09-22T02:59:17.289827+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36044800,"num_objects":550,"num_object_clones":0,"num_object_copies":1650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":550,"num_write_kb":35200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'561","reported_seq":574,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.210636+0000","last_change":"2024-09-22T02:58:55.768855+0000","last_active":"2024-09-22T02:59:17.210636+0000","last_peered":"2024-09-22T02:59:17.210636+0000","last_clean":"2024-09-22T02:59:17.210636+0000","last_became_active":"2024-09-22T02:58:55.764760+0000","last_became_peered":"2024-09-22T02:58:55.764760+0000","last_unstale":"2024-09-22T02:59:17.210636+0000","last_undegraded":"2024-09-22T02:59:17.210636+0000","last_fullsized":"2024-09-22T02:59:17.210636+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36831232,"num_objects":562,"num_object_clones":0,"num_object_copies":1686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":562,"num_write_kb":35968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'573","reported_seq":586,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.280414+0000","last_change":"2024-09-22T02:58:55.757520+0000","last_active":"2024-09-22T02:59:17.280414+0000","last_peered":"2024-09-22T02:59:17.280414+0000","last_clean":"2024-09-22T02:59:17.280414+0000","last_became_active":"2024-09-22T02:58:55.752439+0000","last_became_peered":"2024-09-22T02:58:55.752439+0000","last_unstale":"2024-09-22T02:59:17.280414+0000","last_undegraded":"2024-09-22T02:59:17.280414+0000","last_fullsized":"2024-09-22T02:59:17.280414+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37617664,"num_objects":574,"num_object_clones":0,"num_object_copies":1722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":574,"num_write_kb":36736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'666","reported_seq":679,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:21.299080+0000","last_change":"2024-09-22T02:58:55.754878+0000","last_active":"2024-09-22T02:59:21.299080+0000","last_peered":"2024-09-22T02:59:21.299080+0000","last_clean":"2024-09-22T02:59:21.299080+0000","last_became_active":"2024-09-22T02:58:55.750260+0000","last_became_peered":"2024-09-22T02:58:55.750260+0000","last_unstale":"2024-09-22T02:59:21.299080+0000","last_undegraded":"2024-09-22T02:59:21.299080+0000","last_fullsized":"2024-09-22T02:59:21.299080+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43712512,"num_objects":667,"num_object_clones":0,"num_object_copies":2001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":667,"num_write_kb":42688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":25,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.689703+0000","last_change":"2024-09-22T02:58:47.345653+0000","last_active":"2024-09-22T02:58:56.689703+0000","last_peered":"2024-09-22T02:58:56.689703+0000","last_clean":"2024-09-22T02:58:56.689703+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T02:58:56.689703+0000","last_undegraded":"2024-09-22T02:58:56.689703+0000","last_fullsized":"2024-09-22T02:58:56.689703+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"17'697","reported_seq":710,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:21.215677+0000","last_change":"2024-09-22T02:58:55.766536+0000","last_active":"2024-09-22T02:59:21.215677+0000","last_peered":"2024-09-22T02:59:21.215677+0000","last_clean":"2024-09-22T02:59:21.215677+0000","last_became_active":"2024-09-22T02:58:55.761477+0000","last_became_peered":"2024-09-22T02:58:55.761477+0000","last_unstale":"2024-09-22T02:59:21.215677+0000","last_undegraded":"2024-09-22T02:59:21.215677+0000","last_fullsized":"2024-09-22T02:59:21.215677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":697,"log_dups_size":0,"ondisk_log_size":697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45744128,"num_objects":698,"num_object_clones":0,"num_object_copies":2094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":698,"num_write_kb":44672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":25,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.689190+0000","last_change":"2024-09-22T02:58:47.344427+0000","last_active":"2024-09-22T02:58:56.689190+0000","last_peered":"2024-09-22T02:58:56.689190+0000","last_clean":"2024-09-22T02:58:56.689190+0000","last_became_active":"2024-09-22T02:58:45.366840+0000","last_became_peered":"2024-09-22T02:58:45.366840+0000","last_unstale":"2024-09-22T02:58:56.689190+0000","last_undegraded":"2024-09-22T02:58:56.689190+0000","last_fullsized":"2024-09-22T02:58:56.689190+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'568","reported_seq":581,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.289641+0000","last_change":"2024-09-22T02:58:55.741407+0000","last_active":"2024-09-22T02:59:17.289641+0000","last_peered":"2024-09-22T02:59:17.289641+0000","last_clean":"2024-09-22T02:59:17.289641+0000","last_became_active":"2024-09-22T02:58:55.737022+0000","last_became_peered":"2024-09-22T02:58:55.737022+0000","last_unstale":"2024-09-22T02:59:17.289641+0000","last_undegraded":"2024-09-22T02:59:17.289641+0000","last_fullsized":"2024-09-22T02:59:17.289641+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37289984,"num_objects":569,"num_object_clones":0,"num_object_copies":1707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":569,"num_write_kb":36416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.689097+0000","last_change":"2024-09-22T02:58:47.346629+0000","last_active":"2024-09-22T02:58:56.689097+0000","last_peered":"2024-09-22T02:58:56.689097+0000","last_clean":"2024-09-22T02:58:56.689097+0000","last_became_active":"2024-09-22T02:58:45.371859+0000","last_became_peered":"2024-09-22T02:58:45.371859+0000","last_unstale":"2024-09-22T02:58:56.689097+0000","last_undegraded":"2024-09-22T02:58:56.689097+0000","last_fullsized":"2024-09-22T02:58:56.689097+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"17'588","reported_seq":601,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.248038+0000","last_change":"2024-09-22T02:58:55.752486+0000","last_active":"2024-09-22T02:59:17.248038+0000","last_peered":"2024-09-22T02:59:17.248038+0000","last_clean":"2024-09-22T02:59:17.248038+0000","last_became_active":"2024-09-22T02:58:55.746225+0000","last_became_peered":"2024-09-22T02:58:55.746225+0000","last_unstale":"2024-09-22T02:59:17.248038+0000","last_undegraded":"2024-09-22T02:59:17.248038+0000","last_fullsized":"2024-09-22T02:59:17.248038+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":589,"num_write_kb":37696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":26,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.692727+0000","last_change":"2024-09-22T02:58:47.348407+0000","last_active":"2024-09-22T02:58:56.692727+0000","last_peered":"2024-09-22T02:58:56.692727+0000","last_clean":"2024-09-22T02:58:56.692727+0000","last_became_active":"2024-09-22T02:58:45.369401+0000","last_became_peered":"2024-09-22T02:58:45.369401+0000","last_unstale":"2024-09-22T02:58:56.692727+0000","last_undegraded":"2024-09-22T02:58:56.692727+0000","last_fullsized":"2024-09-22T02:58:56.692727+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":151,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.691481+0000","last_change":"2024-09-22T02:58:45.382298+0000","last_active":"2024-09-22T02:58:56.691481+0000","last_peered":"2024-09-22T02:58:56.691481+0000","last_clean":"2024-09-22T02:58:56.691481+0000","last_became_active":"2024-09-22T02:58:45.373807+0000","last_became_peered":"2024-09-22T02:58:45.373807+0000","last_unstale":"2024-09-22T02:58:56.691481+0000","last_undegraded":"2024-09-22T02:58:56.691481+0000","last_fullsized":"2024-09-22T02:58:56.691481+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'653","reported_seq":666,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.289171+0000","last_change":"2024-09-22T02:58:55.773825+0000","last_active":"2024-09-22T02:59:17.289171+0000","last_peered":"2024-09-22T02:59:17.289171+0000","last_clean":"2024-09-22T02:59:17.289171+0000","last_became_active":"2024-09-22T02:58:55.767555+0000","last_became_peered":"2024-09-22T02:58:55.767555+0000","last_unstale":"2024-09-22T02:59:17.289171+0000","last_undegraded":"2024-09-22T02:59:17.289171+0000","last_fullsized":"2024-09-22T02:59:17.289171+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":653,"log_dups_size":0,"ondisk_log_size":653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42860544,"num_objects":654,"num_object_clones":0,"num_object_copies":1962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":654,"num_write_kb":41856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"17'581","reported_seq":594,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.225933+0000","last_change":"2024-09-22T02:58:55.747999+0000","last_active":"2024-09-22T02:59:17.225933+0000","last_peered":"2024-09-22T02:59:17.225933+0000","last_clean":"2024-09-22T02:59:17.225933+0000","last_became_active":"2024-09-22T02:58:55.743521+0000","last_became_peered":"2024-09-22T02:58:55.743521+0000","last_unstale":"2024-09-22T02:59:17.225933+0000","last_undegraded":"2024-09-22T02:59:17.225933+0000","last_fullsized":"2024-09-22T02:59:17.225933+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":582,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":582,"num_write_kb":37248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":25,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.688762+0000","last_change":"2024-09-22T02:58:47.346116+0000","last_active":"2024-09-22T02:58:56.688762+0000","last_peered":"2024-09-22T02:58:56.688762+0000","last_clean":"2024-09-22T02:58:56.688762+0000","last_became_active":"2024-09-22T02:58:45.374991+0000","last_became_peered":"2024-09-22T02:58:45.374991+0000","last_unstale":"2024-09-22T02:58:56.688762+0000","last_undegraded":"2024-09-22T02:58:56.688762+0000","last_fullsized":"2024-09-22T02:58:56.688762+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"17'569","reported_seq":582,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.179499+0000","last_change":"2024-09-22T02:58:55.759688+0000","last_active":"2024-09-22T02:59:17.179499+0000","last_peered":"2024-09-22T02:59:17.179499+0000","last_clean":"2024-09-22T02:59:17.179499+0000","last_became_active":"2024-09-22T02:58:55.754829+0000","last_became_peered":"2024-09-22T02:58:55.754829+0000","last_unstale":"2024-09-22T02:59:17.179499+0000","last_undegraded":"2024-09-22T02:59:17.179499+0000","last_fullsized":"2024-09-22T02:59:17.179499+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37355520,"num_objects":570,"num_object_clones":0,"num_object_copies":1710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":36480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":25,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.692870+0000","last_change":"2024-09-22T02:58:47.347987+0000","last_active":"2024-09-22T02:58:56.692870+0000","last_peered":"2024-09-22T02:58:56.692870+0000","last_clean":"2024-09-22T02:58:56.692870+0000","last_became_active":"2024-09-22T02:58:45.364066+0000","last_became_peered":"2024-09-22T02:58:45.364066+0000","last_unstale":"2024-09-22T02:58:56.692870+0000","last_undegraded":"2024-09-22T02:58:56.692870+0000","last_fullsized":"2024-09-22T02:58:56.692870+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'557","reported_seq":570,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:17.234805+0000","last_change":"2024-09-22T02:58:55.740822+0000","last_active":"2024-09-22T02:59:17.234805+0000","last_peered":"2024-09-22T02:59:17.234805+0000","last_clean":"2024-09-22T02:59:17.234805+0000","last_became_active":"2024-09-22T02:58:55.733222+0000","last_became_peered":"2024-09-22T02:58:55.733222+0000","last_unstale":"2024-09-22T02:59:17.234805+0000","last_undegraded":"2024-09-22T02:59:17.234805+0000","last_fullsized":"2024-09-22T02:59:17.234805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36569088,"num_objects":558,"num_object_clones":0,"num_object_copies":1674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":558,"num_write_kb":35712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":25,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.692008+0000","last_change":"2024-09-22T02:58:47.347783+0000","last_active":"2024-09-22T02:58:56.692008+0000","last_peered":"2024-09-22T02:58:56.692008+0000","last_clean":"2024-09-22T02:58:56.692008+0000","last_became_active":"2024-09-22T02:58:45.383906+0000","last_became_peered":"2024-09-22T02:58:45.383906+0000","last_unstale":"2024-09-22T02:58:56.692008+0000","last_undegraded":"2024-09-22T02:58:56.692008+0000","last_fullsized":"2024-09-22T02:58:56.692008+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"17'617","reported_seq":630,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:59:21.271811+0000","last_change":"2024-09-22T02:58:55.782093+0000","last_active":"2024-09-22T02:59:21.271811+0000","last_peered":"2024-09-22T02:59:21.271811+0000","last_clean":"2024-09-22T02:59:21.271811+0000","last_became_active":"2024-09-22T02:58:55.771463+0000","last_became_peered":"2024-09-22T02:58:55.771463+0000","last_unstale":"2024-09-22T02:59:21.271811+0000","last_undegraded":"2024-09-22T02:59:21.271811+0000","last_fullsized":"2024-09-22T02:59:21.271811+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":618,"num_object_clones":0,"num_object_copies":1854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":618,"num_write_kb":39552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":25,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-22T02:58:56.691149+0000","last_change":"2024-09-22T02:58:47.347756+0000","last_active":"2024-09-22T02:58:56.691149+0000","last_peered":"2024-09-22T02:58:56.691149+0000","last_clean":"2024-09-22T02:58:56.691149+0000","last_became_active":"2024-09-22T02:58:45.389306+0000","last_became_peered":"2024-09-22T02:58:45.389306+0000","last_unstale":"2024-09-22T02:58:56.691149+0000","last_undegraded":"2024-09-22T02:58:56.691149+0000","last_fullsized":"2024-09-22T02:58:56.691149+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":625606656,"num_objects":9546,"num_object_clones":0,"num_object_copies":28638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9546,"num_write_kb":610944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":117096448,"data_stored":1873543168,"data_compressed":88962526,"data_compressed_allocated":117096448,"data_compressed_original":1873543168,"omap_allocated":0,"internal_metadata":0},"log_size":9530,"ondisk_log_size":9530,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65796,"kb_used_data":30644,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94306044,"statfs":{"total":96636764160,"available":96569389056,"internally_reserved":0,"allocated":31379456,"data_stored":499753603,"data_compressed":23728032,"data_compressed_allocated":31232000,"data_compressed_original":499712000,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":54708,"kb_used_data":23712,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94317132,"statfs":{"total":96636764160,"available":96580743168,"internally_reserved":0,"allocated":24281088,"data_stored":385787043,"data_compressed":18318405,"data_compressed_allocated":24113152,"data_compressed_original":385728512,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55276,"kb_used_data":24284,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94316564,"statfs":{"total":96636764160,"available":96580161536,"internally_reserved":0,"allocated":24866816,"data_stored":395224208,"data_compressed":18766555,"data_compressed_allocated":24702976,"data_compressed_original":395165696,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70752,"kb_used_data":31440,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94301088,"statfs":{"total":96636764160,"available":96564314112,"internally_reserved":0,"allocated":32194560,"data_stored":512402595,"data_compressed":24330739,"data_compressed_allocated":32026624,"data_compressed_original":512344064,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25894912,"data_stored":414318592,"data_compressed":19673308,"data_compressed_allocated":25894912,"data_compressed_original":414318592,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":33251328,"data_stored":532021248,"data_compressed":25262388,"data_compressed_allocated":33251328,"data_compressed_original":532021248,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25243648,"data_stored":403898368,"data_compressed":19178478,"data_compressed_allocated":25243648,"data_compressed_original":403898368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32706560,"data_stored":523304960,"data_compressed":24848352,"data_compressed_allocated":32706560,"data_compressed_original":523304960,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T02:59:23.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 27 16 11762 11746 27.1876 25.625 0.0468048 0.0367474 2024-09-22T02:59:24.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:24.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 28 16 12154 12138 27.0915 24.5 0.0500708 0.0368746 2024-09-22T02:59:24.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:24.786+0000 7ff704618640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:24.793+0000 7ff704618640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:25.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 29 16 12557 12541 27.0258 25.1875 0.0264642 0.0369721 2024-09-22T02:59:26.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 30 16 12961 12945 26.9665 25.25 0.042664 0.0370605 2024-09-22T02:59:26.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:27.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:27.131+0000 7f7ca5517640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:27.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:27.140+0000 7f7c9ffff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:27.753 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 31 15 13343 13328 26.8688 23.9375 0.0345802 0.0371899 2024-09-22T02:59:28.038 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T02:59:28.038 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T02:59:28.038 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T02:59:28.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T02:59:28.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:28.179+0000 7fbee5197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:28.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:28.191+0000 7fbee5197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:28.242 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:28.242 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:28.242 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:28.242 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:28.242 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:28.242 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:28.242 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:28.242 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:28.242 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:28.242 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:28.243 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:28.243 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:28.247 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:28.247 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:28.248 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T02:59:28.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:28.392+0000 7f06e1cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:28.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:28.403+0000 7f06e1cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:28.427 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:28.428 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:28.428 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:28.428 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:28.428 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:28.428 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:28.428 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:28.428 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:28.428 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:28.428 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:28.428 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:28.428 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:28.432 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:28.432 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T02:59:28.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:28.586+0000 7fc2166d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:28.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:28.604+0000 7fc2166d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:28.626 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:28.626 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:28.626 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:28.626 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:28.626 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:28.626 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:28.626 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:28.627 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:28.627 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:28.627 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:28.627 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:28.627 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:28.630 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:28.630 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:28.630 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T02:59:28.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 32 16 13730 13714 26.783 24.125 0.0389233 0.0373091 2024-09-22T02:59:28.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:28.762+0000 7fd35fb57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:28.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:28.773+0000 7fd35fb57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:28.828 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:28.828 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:28.828 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:28.828 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:28.828 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:28.828 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:28.828 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:28.828 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:28.828 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:28.829 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:28.829 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:28.829 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:28.832 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:29.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:29.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:29.481+0000 7f0c62917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:29.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:29.488+0000 7f0c62917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:29.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 33 16 14131 14115 26.7308 25.0625 0.040799 0.0373894 2024-09-22T02:59:30.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 34 16 14518 14502 26.6559 24.1875 0.038059 0.0374787 2024-09-22T02:59:31.655 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:31.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 35 16 14915 14899 26.6032 24.8125 0.0438799 0.0375609 2024-09-22T02:59:31.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:31.790+0000 7f49cfa4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:31.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:31.800+0000 7f49cfa4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:32.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 36 16 15284 15268 26.5048 23.0625 0.0377655 0.0376977 2024-09-22T02:59:33.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 37 16 15678 15662 26.4539 24.625 0.0533195 0.0377729 2024-09-22T02:59:33.834 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T02:59:33.834 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T02:59:33.834 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T02:59:33.834 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T02:59:33.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:33.976+0000 7f9d576fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:33.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:33.988+0000 7f9d576fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:34.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:34.045 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:34.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T02:59:34.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:34.173+0000 7f4c6c6fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:34.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:34.181+0000 7f4c6c6fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:34.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:34.233+0000 7fd204997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:34.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:34.247+0000 7fd204997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:34.327 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:34.327 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:34.327 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:34.327 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:34.328 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:34.328 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:34.328 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:34.328 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:34.328 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:34.328 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:34.328 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:34.328 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:34.331 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:34.331 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T02:59:34.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:34.481+0000 7f418ce58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:34.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:34.494+0000 7f418ce58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:34.524 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:34.525 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:34.525 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:34.525 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:34.525 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:34.525 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:34.525 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:34.525 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:34.525 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:34.525 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:34.525 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:34.525 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:34.528 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:34.528 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T02:59:34.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:34.678+0000 7fa71d2d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:34.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:34.697+0000 7fa71d2d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:34.722 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:34.722 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:34.722 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:34.722 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:34.722 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:34.723 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:34.723 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:34.723 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:34.723 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:34.723 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:34.723 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:34.723 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:34.727 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:34.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 38 16 16033 16017 26.3416 22.1875 0.0417599 0.0379287 2024-09-22T02:59:35.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 39 16 16412 16396 26.2735 23.6875 0.0404152 0.0380304 2024-09-22T02:59:36.385 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:36.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:36.534+0000 7fe87fb57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:36.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:36.544+0000 7fe87fb57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:36.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-22T02:59:36.753912+0000 min lat: 0.0084511 max lat: 0.113602 avg lat: 0.0381424 2024-09-22T02:59:36.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T02:59:36.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 40 16 16780 16764 26.1916 23 0.0513003 0.0381424 2024-09-22T02:59:37.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 41 16 17150 17134 26.1168 23.125 0.0212952 0.0382628 2024-09-22T02:59:38.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 42 16 17529 17513 26.0589 23.6875 0.038021 0.0383507 2024-09-22T02:59:38.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:38.925+0000 7ff8c10d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:38.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:38.931+0000 7ff8c10d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:39.728 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T02:59:39.728 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-22T02:59:39.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 43 16 17891 17875 25.979 22.625 0.0462992 0.038467 2024-09-22T02:59:39.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:39.871+0000 7f4e1e82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:39.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:39.887+0000 7f4e1e82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:40.754 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 44 16 18248 18232 25.8956 22.3125 0.043614 0.0385896 2024-09-22T02:59:41.103 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.0 to 1 (10000) 2024-09-22T02:59:41.125 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-22T02:59:41.197 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:41.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:41.309+0000 7f42d5a6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:41.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:41.328+0000 7f42d5a6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:41.352+0000 7f245c0fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:41.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:41.360+0000 7f245c0fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:41.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 45 16 18591 18575 25.7965 21.4375 0.0426385 0.038743 2024-09-22T02:59:42.112 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.1 to 1 (10000) 2024-09-22T02:59:42.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-22T02:59:42.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:42.270+0000 7ffad266c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:42.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:42.286+0000 7ffad266c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:42.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 46 16 18928 18912 25.6936 21.0625 0.0444746 0.0388914 2024-09-22T02:59:43.128 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.2 to 1 (10000) 2024-09-22T02:59:43.152 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-22T02:59:43.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:43.296+0000 7f0c7524f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:43.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:43.311+0000 7f0c7524f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:43.645 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:43.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 47 16 19290 19274 25.6282 22.625 0.0427202 0.0389962 2024-09-22T02:59:43.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:43.788+0000 7f705b957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:43.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:43.797+0000 7f705b957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:44.128 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.3 to 1 (10000) 2024-09-22T02:59:44.147 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-22T02:59:44.147 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T02:59:44.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:44.323+0000 7f7228317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:44.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:44.336+0000 7f7228317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:44.680 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:59:44.680 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T02:59:44.698 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-22T02:59:44.296349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1224999475,"num_objects":18687,"num_object_clones":0,"num_object_copies":56061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":18822,"num_write_kb":1197608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18767,"ondisk_log_size":18767,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":361920,"kb_used_data":221312,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377125440,"statfs":{"total":386547056640,"available":386176450560,"internally_reserved":0,"allocated":226623488,"data_stored":3615209140,"data_compressed":171663552,"data_compressed_allocated":225951744,"data_compressed_original":3614982144,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":117440512,"num_objects":1792,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1792,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000682"},"pg_stats":[{"pgid":"3.f","version":"19'1158","reported_seq":1177,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.247517+0000","last_change":"2024-09-22T02:58:55.762293+0000","last_active":"2024-09-22T02:59:42.247517+0000","last_peered":"2024-09-22T02:59:42.247517+0000","last_clean":"2024-09-22T02:59:42.247517+0000","last_became_active":"2024-09-22T02:58:55.757845+0000","last_became_peered":"2024-09-22T02:58:55.757845+0000","last_unstale":"2024-09-22T02:59:42.247517+0000","last_undegraded":"2024-09-22T02:59:42.247517+0000","last_fullsized":"2024-09-22T02:59:42.247517+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75956224,"num_objects":1159,"num_object_clones":0,"num_object_copies":3477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1159,"num_write_kb":74176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'1180","reported_seq":1199,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.288976+0000","last_change":"2024-09-22T02:58:55.755437+0000","last_active":"2024-09-22T02:59:42.288976+0000","last_peered":"2024-09-22T02:59:42.288976+0000","last_clean":"2024-09-22T02:59:42.288976+0000","last_became_active":"2024-09-22T02:58:55.750996+0000","last_became_peered":"2024-09-22T02:58:55.750996+0000","last_unstale":"2024-09-22T02:59:42.288976+0000","last_undegraded":"2024-09-22T02:59:42.288976+0000","last_fullsized":"2024-09-22T02:59:42.288976+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1180,"log_dups_size":0,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77398016,"num_objects":1181,"num_object_clones":0,"num_object_copies":3543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1181,"num_write_kb":75584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"18'1179","reported_seq":1195,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.169276+0000","last_change":"2024-09-22T02:58:55.788334+0000","last_active":"2024-09-22T02:59:41.169276+0000","last_peered":"2024-09-22T02:59:41.169276+0000","last_clean":"2024-09-22T02:59:41.169276+0000","last_became_active":"2024-09-22T02:58:55.779349+0000","last_became_peered":"2024-09-22T02:58:55.779349+0000","last_unstale":"2024-09-22T02:59:41.169276+0000","last_undegraded":"2024-09-22T02:59:41.169276+0000","last_fullsized":"2024-09-22T02:59:41.169276+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1179,"log_dups_size":0,"ondisk_log_size":1179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77332480,"num_objects":1180,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1180,"num_write_kb":75520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"19'1118","reported_seq":1137,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.142500+0000","last_change":"2024-09-22T02:58:55.740009+0000","last_active":"2024-09-22T02:59:42.142500+0000","last_peered":"2024-09-22T02:59:42.142500+0000","last_clean":"2024-09-22T02:59:42.142500+0000","last_became_active":"2024-09-22T02:58:55.731779+0000","last_became_peered":"2024-09-22T02:58:55.731779+0000","last_unstale":"2024-09-22T02:59:42.142500+0000","last_undegraded":"2024-09-22T02:59:42.142500+0000","last_fullsized":"2024-09-22T02:59:42.142500+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1118,"log_dups_size":0,"ondisk_log_size":1118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73334784,"num_objects":1119,"num_object_clones":0,"num_object_copies":3357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1119,"num_write_kb":71616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'1140","reported_seq":1159,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.231572+0000","last_change":"2024-09-22T02:58:55.742415+0000","last_active":"2024-09-22T02:59:42.231572+0000","last_peered":"2024-09-22T02:59:42.231572+0000","last_clean":"2024-09-22T02:59:42.231572+0000","last_became_active":"2024-09-22T02:58:55.732221+0000","last_became_peered":"2024-09-22T02:58:55.732221+0000","last_unstale":"2024-09-22T02:59:42.231572+0000","last_undegraded":"2024-09-22T02:59:42.231572+0000","last_fullsized":"2024-09-22T02:59:42.231572+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74776576,"num_objects":1141,"num_object_clones":0,"num_object_copies":3423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1141,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"19'1154","reported_seq":1173,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.235704+0000","last_change":"2024-09-22T02:58:55.768855+0000","last_active":"2024-09-22T02:59:42.235704+0000","last_peered":"2024-09-22T02:59:42.235704+0000","last_clean":"2024-09-22T02:59:42.235704+0000","last_became_active":"2024-09-22T02:58:55.764760+0000","last_became_peered":"2024-09-22T02:58:55.764760+0000","last_unstale":"2024-09-22T02:59:42.235704+0000","last_undegraded":"2024-09-22T02:59:42.235704+0000","last_fullsized":"2024-09-22T02:59:42.235704+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1154,"log_dups_size":0,"ondisk_log_size":1154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75694080,"num_objects":1155,"num_object_clones":0,"num_object_copies":3465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1155,"num_write_kb":73920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"19'1138","reported_seq":1157,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.266099+0000","last_change":"2024-09-22T02:58:55.757520+0000","last_active":"2024-09-22T02:59:42.266099+0000","last_peered":"2024-09-22T02:59:42.266099+0000","last_clean":"2024-09-22T02:59:42.266099+0000","last_became_active":"2024-09-22T02:58:55.752439+0000","last_became_peered":"2024-09-22T02:58:55.752439+0000","last_unstale":"2024-09-22T02:59:42.266099+0000","last_undegraded":"2024-09-22T02:59:42.266099+0000","last_fullsized":"2024-09-22T02:59:42.266099+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1138,"log_dups_size":0,"ondisk_log_size":1138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74645504,"num_objects":1139,"num_object_clones":0,"num_object_copies":3417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1139,"num_write_kb":72896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"18'1184","reported_seq":1200,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.270719+0000","last_change":"2024-09-22T02:58:55.754878+0000","last_active":"2024-09-22T02:59:41.270719+0000","last_peered":"2024-09-22T02:59:41.270719+0000","last_clean":"2024-09-22T02:59:41.270719+0000","last_became_active":"2024-09-22T02:58:55.750260+0000","last_became_peered":"2024-09-22T02:58:55.750260+0000","last_unstale":"2024-09-22T02:59:41.270719+0000","last_undegraded":"2024-09-22T02:59:41.270719+0000","last_fullsized":"2024-09-22T02:59:41.270719+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1184,"log_dups_size":0,"ondisk_log_size":1184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77660160,"num_objects":1185,"num_object_clones":0,"num_object_copies":3555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1185,"num_write_kb":75840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.144178+0000","last_change":"2024-09-22T02:58:47.345653+0000","last_active":"2024-09-22T02:59:42.144178+0000","last_peered":"2024-09-22T02:59:42.144178+0000","last_clean":"2024-09-22T02:59:42.144178+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T02:59:42.144178+0000","last_undegraded":"2024-09-22T02:59:42.144178+0000","last_fullsized":"2024-09-22T02:59:42.144178+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"17'1170","reported_seq":1186,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.154019+0000","last_change":"2024-09-22T02:58:55.766536+0000","last_active":"2024-09-22T02:59:41.154019+0000","last_peered":"2024-09-22T02:59:41.154019+0000","last_clean":"2024-09-22T02:59:41.154019+0000","last_became_active":"2024-09-22T02:58:55.761477+0000","last_became_peered":"2024-09-22T02:58:55.761477+0000","last_unstale":"2024-09-22T02:59:41.154019+0000","last_undegraded":"2024-09-22T02:59:41.154019+0000","last_fullsized":"2024-09-22T02:59:41.154019+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1170,"log_dups_size":0,"ondisk_log_size":1170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76742656,"num_objects":1171,"num_object_clones":0,"num_object_copies":3513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":74944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.127817+0000","last_change":"2024-09-22T02:58:47.344427+0000","last_active":"2024-09-22T02:59:42.127817+0000","last_peered":"2024-09-22T02:59:42.127817+0000","last_clean":"2024-09-22T02:59:42.127817+0000","last_became_active":"2024-09-22T02:58:45.366840+0000","last_became_peered":"2024-09-22T02:58:45.366840+0000","last_unstale":"2024-09-22T02:59:42.127817+0000","last_undegraded":"2024-09-22T02:59:42.127817+0000","last_fullsized":"2024-09-22T02:59:42.127817+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"19'1220","reported_seq":1239,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.162238+0000","last_change":"2024-09-22T02:58:55.741407+0000","last_active":"2024-09-22T02:59:42.162238+0000","last_peered":"2024-09-22T02:59:42.162238+0000","last_clean":"2024-09-22T02:59:42.162238+0000","last_became_active":"2024-09-22T02:58:55.737022+0000","last_became_peered":"2024-09-22T02:58:55.737022+0000","last_unstale":"2024-09-22T02:59:42.162238+0000","last_undegraded":"2024-09-22T02:59:42.162238+0000","last_fullsized":"2024-09-22T02:59:42.162238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1220,"log_dups_size":0,"ondisk_log_size":1220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80019456,"num_objects":1221,"num_object_clones":0,"num_object_copies":3663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1221,"num_write_kb":78144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.143721+0000","last_change":"2024-09-22T02:58:47.346629+0000","last_active":"2024-09-22T02:59:42.143721+0000","last_peered":"2024-09-22T02:59:42.143721+0000","last_clean":"2024-09-22T02:59:42.143721+0000","last_became_active":"2024-09-22T02:58:45.371859+0000","last_became_peered":"2024-09-22T02:58:45.371859+0000","last_unstale":"2024-09-22T02:59:42.143721+0000","last_undegraded":"2024-09-22T02:59:42.143721+0000","last_fullsized":"2024-09-22T02:59:42.143721+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"19'1141","reported_seq":1160,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.217895+0000","last_change":"2024-09-22T02:58:55.752486+0000","last_active":"2024-09-22T02:59:42.217895+0000","last_peered":"2024-09-22T02:59:42.217895+0000","last_clean":"2024-09-22T02:59:42.217895+0000","last_became_active":"2024-09-22T02:58:55.746225+0000","last_became_peered":"2024-09-22T02:58:55.746225+0000","last_unstale":"2024-09-22T02:59:42.217895+0000","last_undegraded":"2024-09-22T02:59:42.217895+0000","last_fullsized":"2024-09-22T02:59:42.217895+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1141,"log_dups_size":0,"ondisk_log_size":1141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74842112,"num_objects":1142,"num_object_clones":0,"num_object_copies":3426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1142,"num_write_kb":73088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.147783+0000","last_change":"2024-09-22T02:58:47.348407+0000","last_active":"2024-09-22T02:59:42.147783+0000","last_peered":"2024-09-22T02:59:42.147783+0000","last_clean":"2024-09-22T02:59:42.147783+0000","last_became_active":"2024-09-22T02:58:45.369401+0000","last_became_peered":"2024-09-22T02:58:45.369401+0000","last_unstale":"2024-09-22T02:59:42.147783+0000","last_undegraded":"2024-09-22T02:59:42.147783+0000","last_fullsized":"2024-09-22T02:59:42.147783+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":157,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.129831+0000","last_change":"2024-09-22T02:58:45.382298+0000","last_active":"2024-09-22T02:59:42.129831+0000","last_peered":"2024-09-22T02:59:42.129831+0000","last_clean":"2024-09-22T02:59:42.129831+0000","last_became_active":"2024-09-22T02:58:45.373807+0000","last_became_peered":"2024-09-22T02:58:45.373807+0000","last_unstale":"2024-09-22T02:59:42.129831+0000","last_undegraded":"2024-09-22T02:59:42.129831+0000","last_fullsized":"2024-09-22T02:59:42.129831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"19'1290","reported_seq":1309,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.247754+0000","last_change":"2024-09-22T02:58:55.773825+0000","last_active":"2024-09-22T02:59:42.247754+0000","last_peered":"2024-09-22T02:59:42.247754+0000","last_clean":"2024-09-22T02:59:42.247754+0000","last_became_active":"2024-09-22T02:58:55.767555+0000","last_became_peered":"2024-09-22T02:58:55.767555+0000","last_unstale":"2024-09-22T02:59:42.247754+0000","last_undegraded":"2024-09-22T02:59:42.247754+0000","last_fullsized":"2024-09-22T02:59:42.247754+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1291,"num_write_kb":82624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"18'1159","reported_seq":1178,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.146032+0000","last_change":"2024-09-22T02:58:55.747999+0000","last_active":"2024-09-22T02:59:42.146032+0000","last_peered":"2024-09-22T02:59:42.146032+0000","last_clean":"2024-09-22T02:59:42.146032+0000","last_became_active":"2024-09-22T02:58:55.743521+0000","last_became_peered":"2024-09-22T02:58:55.743521+0000","last_unstale":"2024-09-22T02:59:42.146032+0000","last_undegraded":"2024-09-22T02:59:42.146032+0000","last_fullsized":"2024-09-22T02:59:42.146032+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1159,"log_dups_size":0,"ondisk_log_size":1159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76021760,"num_objects":1160,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1160,"num_write_kb":74240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.146091+0000","last_change":"2024-09-22T02:58:47.346116+0000","last_active":"2024-09-22T02:59:41.146091+0000","last_peered":"2024-09-22T02:59:41.146091+0000","last_clean":"2024-09-22T02:59:41.146091+0000","last_became_active":"2024-09-22T02:58:45.374991+0000","last_became_peered":"2024-09-22T02:58:45.374991+0000","last_unstale":"2024-09-22T02:59:41.146091+0000","last_undegraded":"2024-09-22T02:59:41.146091+0000","last_fullsized":"2024-09-22T02:59:41.146091+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'1202","reported_seq":1221,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.307089+0000","last_change":"2024-09-22T02:58:55.759688+0000","last_active":"2024-09-22T02:59:42.307089+0000","last_peered":"2024-09-22T02:59:42.307089+0000","last_clean":"2024-09-22T02:59:42.307089+0000","last_became_active":"2024-09-22T02:58:55.754829+0000","last_became_peered":"2024-09-22T02:58:55.754829+0000","last_unstale":"2024-09-22T02:59:42.307089+0000","last_undegraded":"2024-09-22T02:59:42.307089+0000","last_fullsized":"2024-09-22T02:59:42.307089+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1202,"log_dups_size":0,"ondisk_log_size":1202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78839808,"num_objects":1203,"num_object_clones":0,"num_object_copies":3609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1203,"num_write_kb":76992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.146183+0000","last_change":"2024-09-22T02:58:47.347987+0000","last_active":"2024-09-22T02:59:42.146183+0000","last_peered":"2024-09-22T02:59:42.146183+0000","last_clean":"2024-09-22T02:59:42.146183+0000","last_became_active":"2024-09-22T02:58:45.364066+0000","last_became_peered":"2024-09-22T02:58:45.364066+0000","last_unstale":"2024-09-22T02:59:42.146183+0000","last_undegraded":"2024-09-22T02:59:42.146183+0000","last_fullsized":"2024-09-22T02:59:42.146183+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"19'1150","reported_seq":1169,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.247318+0000","last_change":"2024-09-22T02:58:55.740822+0000","last_active":"2024-09-22T02:59:42.247318+0000","last_peered":"2024-09-22T02:59:42.247318+0000","last_clean":"2024-09-22T02:59:42.247318+0000","last_became_active":"2024-09-22T02:58:55.733222+0000","last_became_peered":"2024-09-22T02:58:55.733222+0000","last_unstale":"2024-09-22T02:59:42.247318+0000","last_undegraded":"2024-09-22T02:59:42.247318+0000","last_fullsized":"2024-09-22T02:59:42.247318+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1150,"log_dups_size":0,"ondisk_log_size":1150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75431936,"num_objects":1151,"num_object_clones":0,"num_object_copies":3453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1151,"num_write_kb":73664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.150447+0000","last_change":"2024-09-22T02:58:47.347783+0000","last_active":"2024-09-22T02:59:41.150447+0000","last_peered":"2024-09-22T02:59:41.150447+0000","last_clean":"2024-09-22T02:59:41.150447+0000","last_became_active":"2024-09-22T02:58:45.383906+0000","last_became_peered":"2024-09-22T02:58:45.383906+0000","last_unstale":"2024-09-22T02:59:41.150447+0000","last_undegraded":"2024-09-22T02:59:41.150447+0000","last_fullsized":"2024-09-22T02:59:41.150447+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"18'1084","reported_seq":1100,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.276791+0000","last_change":"2024-09-22T02:58:55.782093+0000","last_active":"2024-09-22T02:59:41.276791+0000","last_peered":"2024-09-22T02:59:41.276791+0000","last_clean":"2024-09-22T02:59:41.276791+0000","last_became_active":"2024-09-22T02:58:55.771463+0000","last_became_peered":"2024-09-22T02:58:55.771463+0000","last_unstale":"2024-09-22T02:59:41.276791+0000","last_undegraded":"2024-09-22T02:59:41.276791+0000","last_fullsized":"2024-09-22T02:59:41.276791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71106560,"num_objects":1085,"num_object_clones":0,"num_object_copies":3255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1085,"num_write_kb":69440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.149921+0000","last_change":"2024-09-22T02:58:47.347756+0000","last_active":"2024-09-22T02:59:41.149921+0000","last_peered":"2024-09-22T02:59:41.149921+0000","last_clean":"2024-09-22T02:59:41.149921+0000","last_became_active":"2024-09-22T02:58:45.389306+0000","last_became_peered":"2024-09-22T02:58:45.389306+0000","last_unstale":"2024-09-22T02:59:41.149921+0000","last_undegraded":"2024-09-22T02:59:41.149921+0000","last_fullsized":"2024-09-22T02:59:41.149921+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1224409088,"num_objects":18683,"num_object_clones":0,"num_object_copies":56049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18683,"num_write_kb":1195712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":229683200,"data_stored":3674931200,"data_compressed":174502067,"data_compressed_allocated":229683200,"data_compressed_original":3674931200,"omap_allocated":0,"internal_metadata":0},"log_size":18667,"ondisk_log_size":18667,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100168,"kb_used_data":65016,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94271672,"statfs":{"total":96636764160,"available":96534192128,"internally_reserved":0,"allocated":66576384,"data_stored":1062776652,"data_compressed":50463223,"data_compressed_allocated":66420736,"data_compressed_original":1062731776,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89632,"kb_used_data":50320,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94282208,"statfs":{"total":96636764160,"available":96544980992,"internally_reserved":0,"allocated":51527680,"data_stored":821604716,"data_compressed":39013205,"data_compressed_allocated":51351552,"data_compressed_original":821542912,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90556,"kb_used_data":51244,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94281284,"statfs":{"total":96636764160,"available":96544034816,"internally_reserved":0,"allocated":52473856,"data_stored":836809049,"data_compressed":39735210,"data_compressed_allocated":52301824,"data_compressed_original":836747264,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81564,"kb_used_data":54732,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94290276,"statfs":{"total":96636764160,"available":96553242624,"internally_reserved":0,"allocated":56045568,"data_stored":894018723,"data_compressed":42451914,"data_compressed_allocated":55877632,"data_compressed_original":893960192,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53170176,"data_stored":850722816,"data_compressed":40396115,"data_compressed_allocated":53170176,"data_compressed_original":850722816,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":56885248,"data_stored":910163968,"data_compressed":43218627,"data_compressed_allocated":56885248,"data_compressed_original":910163968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":52109312,"data_stored":833748992,"data_compressed":39590086,"data_compressed_allocated":52109312,"data_compressed_original":833748992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67518464,"data_stored":1080295424,"data_compressed":51297239,"data_compressed_allocated":67518464,"data_compressed_original":1080295424,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T02:59:44.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T02:59:44.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 48 16 19625 19609 25.5305 20.9375 0.0431994 0.0391472 2024-09-22T02:59:44.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:44.831+0000 7f82cd597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:44.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:44.840+0000 7f82cd597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:45.155 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:59:45.155 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T02:59:45.173 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-22T02:59:44.296349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1224999475,"num_objects":18687,"num_object_clones":0,"num_object_copies":56061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":18822,"num_write_kb":1197608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18767,"ondisk_log_size":18767,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":361920,"kb_used_data":221312,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377125440,"statfs":{"total":386547056640,"available":386176450560,"internally_reserved":0,"allocated":226623488,"data_stored":3615209140,"data_compressed":171663552,"data_compressed_allocated":225951744,"data_compressed_original":3614982144,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":117440512,"num_objects":1792,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1792,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000682"},"pg_stats":[{"pgid":"3.f","version":"19'1158","reported_seq":1177,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.247517+0000","last_change":"2024-09-22T02:58:55.762293+0000","last_active":"2024-09-22T02:59:42.247517+0000","last_peered":"2024-09-22T02:59:42.247517+0000","last_clean":"2024-09-22T02:59:42.247517+0000","last_became_active":"2024-09-22T02:58:55.757845+0000","last_became_peered":"2024-09-22T02:58:55.757845+0000","last_unstale":"2024-09-22T02:59:42.247517+0000","last_undegraded":"2024-09-22T02:59:42.247517+0000","last_fullsized":"2024-09-22T02:59:42.247517+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75956224,"num_objects":1159,"num_object_clones":0,"num_object_copies":3477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1159,"num_write_kb":74176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'1180","reported_seq":1199,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.288976+0000","last_change":"2024-09-22T02:58:55.755437+0000","last_active":"2024-09-22T02:59:42.288976+0000","last_peered":"2024-09-22T02:59:42.288976+0000","last_clean":"2024-09-22T02:59:42.288976+0000","last_became_active":"2024-09-22T02:58:55.750996+0000","last_became_peered":"2024-09-22T02:58:55.750996+0000","last_unstale":"2024-09-22T02:59:42.288976+0000","last_undegraded":"2024-09-22T02:59:42.288976+0000","last_fullsized":"2024-09-22T02:59:42.288976+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1180,"log_dups_size":0,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77398016,"num_objects":1181,"num_object_clones":0,"num_object_copies":3543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1181,"num_write_kb":75584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"18'1179","reported_seq":1195,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.169276+0000","last_change":"2024-09-22T02:58:55.788334+0000","last_active":"2024-09-22T02:59:41.169276+0000","last_peered":"2024-09-22T02:59:41.169276+0000","last_clean":"2024-09-22T02:59:41.169276+0000","last_became_active":"2024-09-22T02:58:55.779349+0000","last_became_peered":"2024-09-22T02:58:55.779349+0000","last_unstale":"2024-09-22T02:59:41.169276+0000","last_undegraded":"2024-09-22T02:59:41.169276+0000","last_fullsized":"2024-09-22T02:59:41.169276+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1179,"log_dups_size":0,"ondisk_log_size":1179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77332480,"num_objects":1180,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1180,"num_write_kb":75520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"19'1118","reported_seq":1137,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.142500+0000","last_change":"2024-09-22T02:58:55.740009+0000","last_active":"2024-09-22T02:59:42.142500+0000","last_peered":"2024-09-22T02:59:42.142500+0000","last_clean":"2024-09-22T02:59:42.142500+0000","last_became_active":"2024-09-22T02:58:55.731779+0000","last_became_peered":"2024-09-22T02:58:55.731779+0000","last_unstale":"2024-09-22T02:59:42.142500+0000","last_undegraded":"2024-09-22T02:59:42.142500+0000","last_fullsized":"2024-09-22T02:59:42.142500+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1118,"log_dups_size":0,"ondisk_log_size":1118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73334784,"num_objects":1119,"num_object_clones":0,"num_object_copies":3357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1119,"num_write_kb":71616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'1140","reported_seq":1159,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.231572+0000","last_change":"2024-09-22T02:58:55.742415+0000","last_active":"2024-09-22T02:59:42.231572+0000","last_peered":"2024-09-22T02:59:42.231572+0000","last_clean":"2024-09-22T02:59:42.231572+0000","last_became_active":"2024-09-22T02:58:55.732221+0000","last_became_peered":"2024-09-22T02:58:55.732221+0000","last_unstale":"2024-09-22T02:59:42.231572+0000","last_undegraded":"2024-09-22T02:59:42.231572+0000","last_fullsized":"2024-09-22T02:59:42.231572+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74776576,"num_objects":1141,"num_object_clones":0,"num_object_copies":3423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1141,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"19'1154","reported_seq":1173,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.235704+0000","last_change":"2024-09-22T02:58:55.768855+0000","last_active":"2024-09-22T02:59:42.235704+0000","last_peered":"2024-09-22T02:59:42.235704+0000","last_clean":"2024-09-22T02:59:42.235704+0000","last_became_active":"2024-09-22T02:58:55.764760+0000","last_became_peered":"2024-09-22T02:58:55.764760+0000","last_unstale":"2024-09-22T02:59:42.235704+0000","last_undegraded":"2024-09-22T02:59:42.235704+0000","last_fullsized":"2024-09-22T02:59:42.235704+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1154,"log_dups_size":0,"ondisk_log_size":1154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75694080,"num_objects":1155,"num_object_clones":0,"num_object_copies":3465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1155,"num_write_kb":73920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"19'1138","reported_seq":1157,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.266099+0000","last_change":"2024-09-22T02:58:55.757520+0000","last_active":"2024-09-22T02:59:42.266099+0000","last_peered":"2024-09-22T02:59:42.266099+0000","last_clean":"2024-09-22T02:59:42.266099+0000","last_became_active":"2024-09-22T02:58:55.752439+0000","last_became_peered":"2024-09-22T02:58:55.752439+0000","last_unstale":"2024-09-22T02:59:42.266099+0000","last_undegraded":"2024-09-22T02:59:42.266099+0000","last_fullsized":"2024-09-22T02:59:42.266099+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1138,"log_dups_size":0,"ondisk_log_size":1138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74645504,"num_objects":1139,"num_object_clones":0,"num_object_copies":3417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1139,"num_write_kb":72896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"18'1184","reported_seq":1200,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.270719+0000","last_change":"2024-09-22T02:58:55.754878+0000","last_active":"2024-09-22T02:59:41.270719+0000","last_peered":"2024-09-22T02:59:41.270719+0000","last_clean":"2024-09-22T02:59:41.270719+0000","last_became_active":"2024-09-22T02:58:55.750260+0000","last_became_peered":"2024-09-22T02:58:55.750260+0000","last_unstale":"2024-09-22T02:59:41.270719+0000","last_undegraded":"2024-09-22T02:59:41.270719+0000","last_fullsized":"2024-09-22T02:59:41.270719+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1184,"log_dups_size":0,"ondisk_log_size":1184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77660160,"num_objects":1185,"num_object_clones":0,"num_object_copies":3555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1185,"num_write_kb":75840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.144178+0000","last_change":"2024-09-22T02:58:47.345653+0000","last_active":"2024-09-22T02:59:42.144178+0000","last_peered":"2024-09-22T02:59:42.144178+0000","last_clean":"2024-09-22T02:59:42.144178+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T02:59:42.144178+0000","last_undegraded":"2024-09-22T02:59:42.144178+0000","last_fullsized":"2024-09-22T02:59:42.144178+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"17'1170","reported_seq":1186,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.154019+0000","last_change":"2024-09-22T02:58:55.766536+0000","last_active":"2024-09-22T02:59:41.154019+0000","last_peered":"2024-09-22T02:59:41.154019+0000","last_clean":"2024-09-22T02:59:41.154019+0000","last_became_active":"2024-09-22T02:58:55.761477+0000","last_became_peered":"2024-09-22T02:58:55.761477+0000","last_unstale":"2024-09-22T02:59:41.154019+0000","last_undegraded":"2024-09-22T02:59:41.154019+0000","last_fullsized":"2024-09-22T02:59:41.154019+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1170,"log_dups_size":0,"ondisk_log_size":1170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76742656,"num_objects":1171,"num_object_clones":0,"num_object_copies":3513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":74944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.127817+0000","last_change":"2024-09-22T02:58:47.344427+0000","last_active":"2024-09-22T02:59:42.127817+0000","last_peered":"2024-09-22T02:59:42.127817+0000","last_clean":"2024-09-22T02:59:42.127817+0000","last_became_active":"2024-09-22T02:58:45.366840+0000","last_became_peered":"2024-09-22T02:58:45.366840+0000","last_unstale":"2024-09-22T02:59:42.127817+0000","last_undegraded":"2024-09-22T02:59:42.127817+0000","last_fullsized":"2024-09-22T02:59:42.127817+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"19'1220","reported_seq":1239,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.162238+0000","last_change":"2024-09-22T02:58:55.741407+0000","last_active":"2024-09-22T02:59:42.162238+0000","last_peered":"2024-09-22T02:59:42.162238+0000","last_clean":"2024-09-22T02:59:42.162238+0000","last_became_active":"2024-09-22T02:58:55.737022+0000","last_became_peered":"2024-09-22T02:58:55.737022+0000","last_unstale":"2024-09-22T02:59:42.162238+0000","last_undegraded":"2024-09-22T02:59:42.162238+0000","last_fullsized":"2024-09-22T02:59:42.162238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1220,"log_dups_size":0,"ondisk_log_size":1220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80019456,"num_objects":1221,"num_object_clones":0,"num_object_copies":3663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1221,"num_write_kb":78144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.143721+0000","last_change":"2024-09-22T02:58:47.346629+0000","last_active":"2024-09-22T02:59:42.143721+0000","last_peered":"2024-09-22T02:59:42.143721+0000","last_clean":"2024-09-22T02:59:42.143721+0000","last_became_active":"2024-09-22T02:58:45.371859+0000","last_became_peered":"2024-09-22T02:58:45.371859+0000","last_unstale":"2024-09-22T02:59:42.143721+0000","last_undegraded":"2024-09-22T02:59:42.143721+0000","last_fullsized":"2024-09-22T02:59:42.143721+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"19'1141","reported_seq":1160,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.217895+0000","last_change":"2024-09-22T02:58:55.752486+0000","last_active":"2024-09-22T02:59:42.217895+0000","last_peered":"2024-09-22T02:59:42.217895+0000","last_clean":"2024-09-22T02:59:42.217895+0000","last_became_active":"2024-09-22T02:58:55.746225+0000","last_became_peered":"2024-09-22T02:58:55.746225+0000","last_unstale":"2024-09-22T02:59:42.217895+0000","last_undegraded":"2024-09-22T02:59:42.217895+0000","last_fullsized":"2024-09-22T02:59:42.217895+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1141,"log_dups_size":0,"ondisk_log_size":1141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74842112,"num_objects":1142,"num_object_clones":0,"num_object_copies":3426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1142,"num_write_kb":73088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.147783+0000","last_change":"2024-09-22T02:58:47.348407+0000","last_active":"2024-09-22T02:59:42.147783+0000","last_peered":"2024-09-22T02:59:42.147783+0000","last_clean":"2024-09-22T02:59:42.147783+0000","last_became_active":"2024-09-22T02:58:45.369401+0000","last_became_peered":"2024-09-22T02:58:45.369401+0000","last_unstale":"2024-09-22T02:59:42.147783+0000","last_undegraded":"2024-09-22T02:59:42.147783+0000","last_fullsized":"2024-09-22T02:59:42.147783+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":157,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.129831+0000","last_change":"2024-09-22T02:58:45.382298+0000","last_active":"2024-09-22T02:59:42.129831+0000","last_peered":"2024-09-22T02:59:42.129831+0000","last_clean":"2024-09-22T02:59:42.129831+0000","last_became_active":"2024-09-22T02:58:45.373807+0000","last_became_peered":"2024-09-22T02:58:45.373807+0000","last_unstale":"2024-09-22T02:59:42.129831+0000","last_undegraded":"2024-09-22T02:59:42.129831+0000","last_fullsized":"2024-09-22T02:59:42.129831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"19'1290","reported_seq":1309,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.247754+0000","last_change":"2024-09-22T02:58:55.773825+0000","last_active":"2024-09-22T02:59:42.247754+0000","last_peered":"2024-09-22T02:59:42.247754+0000","last_clean":"2024-09-22T02:59:42.247754+0000","last_became_active":"2024-09-22T02:58:55.767555+0000","last_became_peered":"2024-09-22T02:58:55.767555+0000","last_unstale":"2024-09-22T02:59:42.247754+0000","last_undegraded":"2024-09-22T02:59:42.247754+0000","last_fullsized":"2024-09-22T02:59:42.247754+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1291,"num_write_kb":82624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"18'1159","reported_seq":1178,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.146032+0000","last_change":"2024-09-22T02:58:55.747999+0000","last_active":"2024-09-22T02:59:42.146032+0000","last_peered":"2024-09-22T02:59:42.146032+0000","last_clean":"2024-09-22T02:59:42.146032+0000","last_became_active":"2024-09-22T02:58:55.743521+0000","last_became_peered":"2024-09-22T02:58:55.743521+0000","last_unstale":"2024-09-22T02:59:42.146032+0000","last_undegraded":"2024-09-22T02:59:42.146032+0000","last_fullsized":"2024-09-22T02:59:42.146032+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1159,"log_dups_size":0,"ondisk_log_size":1159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76021760,"num_objects":1160,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1160,"num_write_kb":74240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.146091+0000","last_change":"2024-09-22T02:58:47.346116+0000","last_active":"2024-09-22T02:59:41.146091+0000","last_peered":"2024-09-22T02:59:41.146091+0000","last_clean":"2024-09-22T02:59:41.146091+0000","last_became_active":"2024-09-22T02:58:45.374991+0000","last_became_peered":"2024-09-22T02:58:45.374991+0000","last_unstale":"2024-09-22T02:59:41.146091+0000","last_undegraded":"2024-09-22T02:59:41.146091+0000","last_fullsized":"2024-09-22T02:59:41.146091+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'1202","reported_seq":1221,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.307089+0000","last_change":"2024-09-22T02:58:55.759688+0000","last_active":"2024-09-22T02:59:42.307089+0000","last_peered":"2024-09-22T02:59:42.307089+0000","last_clean":"2024-09-22T02:59:42.307089+0000","last_became_active":"2024-09-22T02:58:55.754829+0000","last_became_peered":"2024-09-22T02:58:55.754829+0000","last_unstale":"2024-09-22T02:59:42.307089+0000","last_undegraded":"2024-09-22T02:59:42.307089+0000","last_fullsized":"2024-09-22T02:59:42.307089+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1202,"log_dups_size":0,"ondisk_log_size":1202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78839808,"num_objects":1203,"num_object_clones":0,"num_object_copies":3609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1203,"num_write_kb":76992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.146183+0000","last_change":"2024-09-22T02:58:47.347987+0000","last_active":"2024-09-22T02:59:42.146183+0000","last_peered":"2024-09-22T02:59:42.146183+0000","last_clean":"2024-09-22T02:59:42.146183+0000","last_became_active":"2024-09-22T02:58:45.364066+0000","last_became_peered":"2024-09-22T02:58:45.364066+0000","last_unstale":"2024-09-22T02:59:42.146183+0000","last_undegraded":"2024-09-22T02:59:42.146183+0000","last_fullsized":"2024-09-22T02:59:42.146183+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"19'1150","reported_seq":1169,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-22T02:59:42.247318+0000","last_change":"2024-09-22T02:58:55.740822+0000","last_active":"2024-09-22T02:59:42.247318+0000","last_peered":"2024-09-22T02:59:42.247318+0000","last_clean":"2024-09-22T02:59:42.247318+0000","last_became_active":"2024-09-22T02:58:55.733222+0000","last_became_peered":"2024-09-22T02:58:55.733222+0000","last_unstale":"2024-09-22T02:59:42.247318+0000","last_undegraded":"2024-09-22T02:59:42.247318+0000","last_fullsized":"2024-09-22T02:59:42.247318+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1150,"log_dups_size":0,"ondisk_log_size":1150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75431936,"num_objects":1151,"num_object_clones":0,"num_object_copies":3453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1151,"num_write_kb":73664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.150447+0000","last_change":"2024-09-22T02:58:47.347783+0000","last_active":"2024-09-22T02:59:41.150447+0000","last_peered":"2024-09-22T02:59:41.150447+0000","last_clean":"2024-09-22T02:59:41.150447+0000","last_became_active":"2024-09-22T02:58:45.383906+0000","last_became_peered":"2024-09-22T02:58:45.383906+0000","last_unstale":"2024-09-22T02:59:41.150447+0000","last_undegraded":"2024-09-22T02:59:41.150447+0000","last_fullsized":"2024-09-22T02:59:41.150447+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"18'1084","reported_seq":1100,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.276791+0000","last_change":"2024-09-22T02:58:55.782093+0000","last_active":"2024-09-22T02:59:41.276791+0000","last_peered":"2024-09-22T02:59:41.276791+0000","last_clean":"2024-09-22T02:59:41.276791+0000","last_became_active":"2024-09-22T02:58:55.771463+0000","last_became_peered":"2024-09-22T02:58:55.771463+0000","last_unstale":"2024-09-22T02:59:41.276791+0000","last_undegraded":"2024-09-22T02:59:41.276791+0000","last_fullsized":"2024-09-22T02:59:41.276791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71106560,"num_objects":1085,"num_object_clones":0,"num_object_copies":3255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1085,"num_write_kb":69440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-22T02:59:41.149921+0000","last_change":"2024-09-22T02:58:47.347756+0000","last_active":"2024-09-22T02:59:41.149921+0000","last_peered":"2024-09-22T02:59:41.149921+0000","last_clean":"2024-09-22T02:59:41.149921+0000","last_became_active":"2024-09-22T02:58:45.389306+0000","last_became_peered":"2024-09-22T02:58:45.389306+0000","last_unstale":"2024-09-22T02:59:41.149921+0000","last_undegraded":"2024-09-22T02:59:41.149921+0000","last_fullsized":"2024-09-22T02:59:41.149921+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1224409088,"num_objects":18683,"num_object_clones":0,"num_object_copies":56049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18683,"num_write_kb":1195712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":229683200,"data_stored":3674931200,"data_compressed":174502067,"data_compressed_allocated":229683200,"data_compressed_original":3674931200,"omap_allocated":0,"internal_metadata":0},"log_size":18667,"ondisk_log_size":18667,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100168,"kb_used_data":65016,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94271672,"statfs":{"total":96636764160,"available":96534192128,"internally_reserved":0,"allocated":66576384,"data_stored":1062776652,"data_compressed":50463223,"data_compressed_allocated":66420736,"data_compressed_original":1062731776,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89632,"kb_used_data":50320,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94282208,"statfs":{"total":96636764160,"available":96544980992,"internally_reserved":0,"allocated":51527680,"data_stored":821604716,"data_compressed":39013205,"data_compressed_allocated":51351552,"data_compressed_original":821542912,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90556,"kb_used_data":51244,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94281284,"statfs":{"total":96636764160,"available":96544034816,"internally_reserved":0,"allocated":52473856,"data_stored":836809049,"data_compressed":39735210,"data_compressed_allocated":52301824,"data_compressed_original":836747264,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81564,"kb_used_data":54732,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94290276,"statfs":{"total":96636764160,"available":96553242624,"internally_reserved":0,"allocated":56045568,"data_stored":894018723,"data_compressed":42451914,"data_compressed_allocated":55877632,"data_compressed_original":893960192,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53170176,"data_stored":850722816,"data_compressed":40396115,"data_compressed_allocated":53170176,"data_compressed_original":850722816,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":56885248,"data_stored":910163968,"data_compressed":43218627,"data_compressed_allocated":56885248,"data_compressed_original":910163968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":52109312,"data_stored":833748992,"data_compressed":39590086,"data_compressed_allocated":52109312,"data_compressed_original":833748992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67518464,"data_stored":1080295424,"data_compressed":51297239,"data_compressed_allocated":67518464,"data_compressed_original":1080295424,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T02:59:45.174 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-22T02:59:45.175 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T02:59:45.175 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T02:59:45.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T02:59:45.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:45.319+0000 7f26d62b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:45.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:45.332+0000 7f26d62b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:45.426 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:45.426 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:45.426 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:45.431 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:45.431 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T02:59:45.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:45.592+0000 7f7bb4223640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:45.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:45.602+0000 7f7bb4223640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:45.675 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:45.676 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:45.676 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:45.676 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:45.676 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:45.676 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:45.676 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:45.676 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:45.676 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:45.676 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:45.676 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:45.676 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:45.680 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:45.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T02:59:45.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 49 16 19979 19963 25.4609 22.125 0.0355925 0.0392558 2024-09-22T02:59:45.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:45.829+0000 7f15b4658640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:45.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:45.846+0000 7f15b4658640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:45.904 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:45.904 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:45.904 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:45.904 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:45.904 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:45.905 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:45.905 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:45.905 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:45.905 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:45.905 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:45.905 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:45.905 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:45.910 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:45.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T02:59:46.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:46.063+0000 7f6c85c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:46.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:46.073+0000 7f6c85c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:46.101 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T02:59:46.101 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T02:59:46.101 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T02:59:46.101 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T02:59:46.102 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T02:59:46.102 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T02:59:46.102 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T02:59:46.102 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T02:59:46.102 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T02:59:46.102 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T02:59:46.102 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T02:59:46.102 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T02:59:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T02:59:46.106 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T02:59:46.136 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:46.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:46.278+0000 7f534d46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:46.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:46.285+0000 7f534d46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:46.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 50 16 20310 20294 25.3654 20.6875 0.0591262 0.0394008 2024-09-22T02:59:47.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 51 16 20668 20652 25.3068 22.375 0.0577094 0.0394883 2024-09-22T02:59:48.588 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:48.725+0000 7fc174917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:48.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:48.732+0000 7fc174917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:48.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 52 16 21017 21001 25.2395 21.8125 0.0291075 0.0395947 2024-09-22T02:59:49.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 53 16 21384 21368 25.1961 22.9375 0.0488192 0.0396684 2024-09-22T02:59:50.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 54 16 21742 21726 25.1438 22.375 0.0382339 0.0397483 2024-09-22T02:59:51.106 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T02:59:51.107 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T02:59:51.107 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T02:59:51.152 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:51.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:51.261+0000 7efe0ead7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:51.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:51.276+0000 7efe0d875640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:51.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:51.294+0000 7fd424b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:51.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:51.306+0000 7fd424b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:51.595 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T02:59:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T02:59:51.615 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2024-09-22T02:59:50.297386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1339032115,"num_objects":20427,"num_object_clones":0,"num_object_copies":61281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":20562,"num_write_kb":1308968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20507,"ondisk_log_size":20507,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":384044,"kb_used_data":242024,"kb_used_omap":0,"kb_used_meta":141952,"kb_avail":377103316,"statfs":{"total":386547056640,"available":386153795584,"internally_reserved":0,"allocated":247832576,"data_stored":3952893161,"data_compressed":187696576,"data_compressed_allocated":247054336,"data_compressed_original":3952623616,"omap_allocated":0,"internal_metadata":145358848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":201588736,"num_objects":3076,"num_object_clones":0,"num_object_copies":9228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3076,"num_write_kb":196864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001370"},"pg_stats":[{"pgid":"3.f","version":"21'1271","reported_seq":1296,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.267702+0000","last_change":"2024-09-22T02:58:55.762293+0000","last_active":"2024-09-22T02:59:47.267702+0000","last_peered":"2024-09-22T02:59:47.267702+0000","last_clean":"2024-09-22T02:59:47.267702+0000","last_became_active":"2024-09-22T02:58:55.757845+0000","last_became_peered":"2024-09-22T02:58:55.757845+0000","last_unstale":"2024-09-22T02:59:47.267702+0000","last_undegraded":"2024-09-22T02:59:47.267702+0000","last_fullsized":"2024-09-22T02:59:47.267702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1271,"log_dups_size":0,"ondisk_log_size":1271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83361792,"num_objects":1272,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1272,"num_write_kb":81408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"21'1291","reported_seq":1316,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.044522+0000","last_change":"2024-09-22T02:58:55.755437+0000","last_active":"2024-09-22T02:59:47.044522+0000","last_peered":"2024-09-22T02:59:47.044522+0000","last_clean":"2024-09-22T02:59:47.044522+0000","last_became_active":"2024-09-22T02:58:55.750996+0000","last_became_peered":"2024-09-22T02:58:55.750996+0000","last_unstale":"2024-09-22T02:59:47.044522+0000","last_undegraded":"2024-09-22T02:59:47.044522+0000","last_fullsized":"2024-09-22T02:59:47.044522+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84672512,"num_objects":1292,"num_object_clones":0,"num_object_copies":3876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1292,"num_write_kb":82688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'1288","reported_seq":1313,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:46.293820+0000","last_change":"2024-09-22T02:58:55.788334+0000","last_active":"2024-09-22T02:59:46.293820+0000","last_peered":"2024-09-22T02:59:46.293820+0000","last_clean":"2024-09-22T02:59:46.293820+0000","last_became_active":"2024-09-22T02:58:55.779349+0000","last_became_peered":"2024-09-22T02:58:55.779349+0000","last_unstale":"2024-09-22T02:59:46.293820+0000","last_undegraded":"2024-09-22T02:59:46.293820+0000","last_fullsized":"2024-09-22T02:59:46.293820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1288,"log_dups_size":0,"ondisk_log_size":1288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84475904,"num_objects":1289,"num_object_clones":0,"num_object_copies":3867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1289,"num_write_kb":82496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"21'1210","reported_seq":1235,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.183385+0000","last_change":"2024-09-22T02:58:55.740009+0000","last_active":"2024-09-22T02:59:47.183385+0000","last_peered":"2024-09-22T02:59:47.183385+0000","last_clean":"2024-09-22T02:59:47.183385+0000","last_became_active":"2024-09-22T02:58:55.731779+0000","last_became_peered":"2024-09-22T02:58:55.731779+0000","last_unstale":"2024-09-22T02:59:47.183385+0000","last_undegraded":"2024-09-22T02:59:47.183385+0000","last_fullsized":"2024-09-22T02:59:47.183385+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1210,"log_dups_size":0,"ondisk_log_size":1210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79364096,"num_objects":1211,"num_object_clones":0,"num_object_copies":3633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1211,"num_write_kb":77504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'1260","reported_seq":1285,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.244606+0000","last_change":"2024-09-22T02:58:55.742415+0000","last_active":"2024-09-22T02:59:47.244606+0000","last_peered":"2024-09-22T02:59:47.244606+0000","last_clean":"2024-09-22T02:59:47.244606+0000","last_became_active":"2024-09-22T02:58:55.732221+0000","last_became_peered":"2024-09-22T02:58:55.732221+0000","last_unstale":"2024-09-22T02:59:47.244606+0000","last_undegraded":"2024-09-22T02:59:47.244606+0000","last_fullsized":"2024-09-22T02:59:47.244606+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1260,"log_dups_size":0,"ondisk_log_size":1260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82640896,"num_objects":1261,"num_object_clones":0,"num_object_copies":3783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1261,"num_write_kb":80704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"21'1247","reported_seq":1272,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.240665+0000","last_change":"2024-09-22T02:58:55.768855+0000","last_active":"2024-09-22T02:59:47.240665+0000","last_peered":"2024-09-22T02:59:47.240665+0000","last_clean":"2024-09-22T02:59:47.240665+0000","last_became_active":"2024-09-22T02:58:55.764760+0000","last_became_peered":"2024-09-22T02:58:55.764760+0000","last_unstale":"2024-09-22T02:59:47.240665+0000","last_undegraded":"2024-09-22T02:59:47.240665+0000","last_fullsized":"2024-09-22T02:59:47.240665+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1247,"log_dups_size":0,"ondisk_log_size":1247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81788928,"num_objects":1248,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1248,"num_write_kb":79872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"21'1254","reported_seq":1279,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.263753+0000","last_change":"2024-09-22T02:58:55.757520+0000","last_active":"2024-09-22T02:59:47.263753+0000","last_peered":"2024-09-22T02:59:47.263753+0000","last_clean":"2024-09-22T02:59:47.263753+0000","last_became_active":"2024-09-22T02:58:55.752439+0000","last_became_peered":"2024-09-22T02:58:55.752439+0000","last_unstale":"2024-09-22T02:59:47.263753+0000","last_undegraded":"2024-09-22T02:59:47.263753+0000","last_fullsized":"2024-09-22T02:59:47.263753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1254,"log_dups_size":0,"ondisk_log_size":1254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82247680,"num_objects":1255,"num_object_clones":0,"num_object_copies":3765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1255,"num_write_kb":80320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"21'1296","reported_seq":1321,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:46.207035+0000","last_change":"2024-09-22T02:58:55.754878+0000","last_active":"2024-09-22T02:59:46.207035+0000","last_peered":"2024-09-22T02:59:46.207035+0000","last_clean":"2024-09-22T02:59:46.207035+0000","last_became_active":"2024-09-22T02:58:55.750260+0000","last_became_peered":"2024-09-22T02:58:55.750260+0000","last_unstale":"2024-09-22T02:59:46.207035+0000","last_undegraded":"2024-09-22T02:59:46.207035+0000","last_fullsized":"2024-09-22T02:59:46.207035+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1296,"log_dups_size":0,"ondisk_log_size":1296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85000192,"num_objects":1297,"num_object_clones":0,"num_object_copies":3891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1297,"num_write_kb":83008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":37,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:44.156623+0000","last_change":"2024-09-22T02:58:47.345653+0000","last_active":"2024-09-22T02:59:44.156623+0000","last_peered":"2024-09-22T02:59:44.156623+0000","last_clean":"2024-09-22T02:59:44.156623+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T02:59:44.156623+0000","last_undegraded":"2024-09-22T02:59:44.156623+0000","last_fullsized":"2024-09-22T02:59:44.156623+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"21'1275","reported_seq":1300,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:46.270079+0000","last_change":"2024-09-22T02:58:55.766536+0000","last_active":"2024-09-22T02:59:46.270079+0000","last_peered":"2024-09-22T02:59:46.270079+0000","last_clean":"2024-09-22T02:59:46.270079+0000","last_became_active":"2024-09-22T02:58:55.761477+0000","last_became_peered":"2024-09-22T02:58:55.761477+0000","last_unstale":"2024-09-22T02:59:46.270079+0000","last_undegraded":"2024-09-22T02:59:46.270079+0000","last_fullsized":"2024-09-22T02:59:46.270079+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1275,"log_dups_size":0,"ondisk_log_size":1275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83623936,"num_objects":1276,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1276,"num_write_kb":81664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:44.151086+0000","last_change":"2024-09-22T02:58:47.344427+0000","last_active":"2024-09-22T02:59:44.151086+0000","last_peered":"2024-09-22T02:59:44.151086+0000","last_clean":"2024-09-22T02:59:44.151086+0000","last_became_active":"2024-09-22T02:58:45.366840+0000","last_became_peered":"2024-09-22T02:58:45.366840+0000","last_unstale":"2024-09-22T02:59:44.151086+0000","last_undegraded":"2024-09-22T02:59:44.151086+0000","last_fullsized":"2024-09-22T02:59:44.151086+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"21'1325","reported_seq":1350,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.190998+0000","last_change":"2024-09-22T02:58:55.741407+0000","last_active":"2024-09-22T02:59:47.190998+0000","last_peered":"2024-09-22T02:59:47.190998+0000","last_clean":"2024-09-22T02:59:47.190998+0000","last_became_active":"2024-09-22T02:58:55.737022+0000","last_became_peered":"2024-09-22T02:58:55.737022+0000","last_unstale":"2024-09-22T02:59:47.190998+0000","last_undegraded":"2024-09-22T02:59:47.190998+0000","last_fullsized":"2024-09-22T02:59:47.190998+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1325,"log_dups_size":0,"ondisk_log_size":1325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86900736,"num_objects":1326,"num_object_clones":0,"num_object_copies":3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1326,"num_write_kb":84864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":39,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:44.138767+0000","last_change":"2024-09-22T02:58:47.346629+0000","last_active":"2024-09-22T02:59:44.138767+0000","last_peered":"2024-09-22T02:59:44.138767+0000","last_clean":"2024-09-22T02:59:44.138767+0000","last_became_active":"2024-09-22T02:58:45.371859+0000","last_became_peered":"2024-09-22T02:58:45.371859+0000","last_unstale":"2024-09-22T02:59:44.138767+0000","last_undegraded":"2024-09-22T02:59:44.138767+0000","last_fullsized":"2024-09-22T02:59:44.138767+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"21'1262","reported_seq":1287,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.266698+0000","last_change":"2024-09-22T02:58:55.752486+0000","last_active":"2024-09-22T02:59:47.266698+0000","last_peered":"2024-09-22T02:59:47.266698+0000","last_clean":"2024-09-22T02:59:47.266698+0000","last_became_active":"2024-09-22T02:58:55.746225+0000","last_became_peered":"2024-09-22T02:58:55.746225+0000","last_unstale":"2024-09-22T02:59:47.266698+0000","last_undegraded":"2024-09-22T02:59:47.266698+0000","last_fullsized":"2024-09-22T02:59:47.266698+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1262,"log_dups_size":0,"ondisk_log_size":1262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82771968,"num_objects":1263,"num_object_clones":0,"num_object_copies":3789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1263,"num_write_kb":80832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":38,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:44.152363+0000","last_change":"2024-09-22T02:58:47.348407+0000","last_active":"2024-09-22T02:59:44.152363+0000","last_peered":"2024-09-22T02:59:44.152363+0000","last_clean":"2024-09-22T02:59:44.152363+0000","last_became_active":"2024-09-22T02:58:45.369401+0000","last_became_peered":"2024-09-22T02:58:45.369401+0000","last_unstale":"2024-09-22T02:59:44.152363+0000","last_undegraded":"2024-09-22T02:59:44.152363+0000","last_fullsized":"2024-09-22T02:59:44.152363+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":163,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:44.152606+0000","last_change":"2024-09-22T02:58:45.382298+0000","last_active":"2024-09-22T02:59:44.152606+0000","last_peered":"2024-09-22T02:59:44.152606+0000","last_clean":"2024-09-22T02:59:44.152606+0000","last_became_active":"2024-09-22T02:58:45.373807+0000","last_became_peered":"2024-09-22T02:58:45.373807+0000","last_unstale":"2024-09-22T02:59:44.152606+0000","last_undegraded":"2024-09-22T02:59:44.152606+0000","last_fullsized":"2024-09-22T02:59:44.152606+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"21'1405","reported_seq":1430,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.218926+0000","last_change":"2024-09-22T02:58:55.773825+0000","last_active":"2024-09-22T02:59:47.218926+0000","last_peered":"2024-09-22T02:59:47.218926+0000","last_clean":"2024-09-22T02:59:47.218926+0000","last_became_active":"2024-09-22T02:58:55.767555+0000","last_became_peered":"2024-09-22T02:58:55.767555+0000","last_unstale":"2024-09-22T02:59:47.218926+0000","last_undegraded":"2024-09-22T02:59:47.218926+0000","last_fullsized":"2024-09-22T02:59:47.218926+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1405,"log_dups_size":0,"ondisk_log_size":1405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92143616,"num_objects":1406,"num_object_clones":0,"num_object_copies":4218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1406,"num_write_kb":89984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"21'1279","reported_seq":1304,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.276383+0000","last_change":"2024-09-22T02:58:55.747999+0000","last_active":"2024-09-22T02:59:47.276383+0000","last_peered":"2024-09-22T02:59:47.276383+0000","last_clean":"2024-09-22T02:59:47.276383+0000","last_became_active":"2024-09-22T02:58:55.743521+0000","last_became_peered":"2024-09-22T02:58:55.743521+0000","last_unstale":"2024-09-22T02:59:47.276383+0000","last_undegraded":"2024-09-22T02:59:47.276383+0000","last_fullsized":"2024-09-22T02:59:47.276383+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1279,"log_dups_size":0,"ondisk_log_size":1279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":1280,"num_object_clones":0,"num_object_copies":3840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1280,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:44.155988+0000","last_change":"2024-09-22T02:58:47.346116+0000","last_active":"2024-09-22T02:59:44.155988+0000","last_peered":"2024-09-22T02:59:44.155988+0000","last_clean":"2024-09-22T02:59:44.155988+0000","last_became_active":"2024-09-22T02:58:45.374991+0000","last_became_peered":"2024-09-22T02:58:45.374991+0000","last_unstale":"2024-09-22T02:59:44.155988+0000","last_undegraded":"2024-09-22T02:59:44.155988+0000","last_fullsized":"2024-09-22T02:59:44.155988+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"21'1326","reported_seq":1351,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.272053+0000","last_change":"2024-09-22T02:58:55.759688+0000","last_active":"2024-09-22T02:59:47.272053+0000","last_peered":"2024-09-22T02:59:47.272053+0000","last_clean":"2024-09-22T02:59:47.272053+0000","last_became_active":"2024-09-22T02:58:55.754829+0000","last_became_peered":"2024-09-22T02:58:55.754829+0000","last_unstale":"2024-09-22T02:59:47.272053+0000","last_undegraded":"2024-09-22T02:59:47.272053+0000","last_fullsized":"2024-09-22T02:59:47.272053+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1326,"log_dups_size":0,"ondisk_log_size":1326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86966272,"num_objects":1327,"num_object_clones":0,"num_object_copies":3981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1327,"num_write_kb":84928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:44.140753+0000","last_change":"2024-09-22T02:58:47.347987+0000","last_active":"2024-09-22T02:59:44.140753+0000","last_peered":"2024-09-22T02:59:44.140753+0000","last_clean":"2024-09-22T02:59:44.140753+0000","last_became_active":"2024-09-22T02:58:45.364066+0000","last_became_peered":"2024-09-22T02:58:45.364066+0000","last_unstale":"2024-09-22T02:59:44.140753+0000","last_undegraded":"2024-09-22T02:59:44.140753+0000","last_fullsized":"2024-09-22T02:59:44.140753+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"21'1238","reported_seq":1263,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:47.228315+0000","last_change":"2024-09-22T02:58:55.740822+0000","last_active":"2024-09-22T02:59:47.228315+0000","last_peered":"2024-09-22T02:59:47.228315+0000","last_clean":"2024-09-22T02:59:47.228315+0000","last_became_active":"2024-09-22T02:58:55.733222+0000","last_became_peered":"2024-09-22T02:58:55.733222+0000","last_unstale":"2024-09-22T02:59:47.228315+0000","last_undegraded":"2024-09-22T02:59:47.228315+0000","last_fullsized":"2024-09-22T02:59:47.228315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1238,"log_dups_size":0,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81199104,"num_objects":1239,"num_object_clones":0,"num_object_copies":3717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1239,"num_write_kb":79296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:44.159204+0000","last_change":"2024-09-22T02:58:47.347783+0000","last_active":"2024-09-22T02:59:44.159204+0000","last_peered":"2024-09-22T02:59:44.159204+0000","last_clean":"2024-09-22T02:59:44.159204+0000","last_became_active":"2024-09-22T02:58:45.383906+0000","last_became_peered":"2024-09-22T02:58:45.383906+0000","last_unstale":"2024-09-22T02:59:44.159204+0000","last_undegraded":"2024-09-22T02:59:44.159204+0000","last_fullsized":"2024-09-22T02:59:44.159204+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"21'1180","reported_seq":1205,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:46.124716+0000","last_change":"2024-09-22T02:58:55.782093+0000","last_active":"2024-09-22T02:59:46.124716+0000","last_peered":"2024-09-22T02:59:46.124716+0000","last_clean":"2024-09-22T02:59:46.124716+0000","last_became_active":"2024-09-22T02:58:55.771463+0000","last_became_peered":"2024-09-22T02:58:55.771463+0000","last_unstale":"2024-09-22T02:59:46.124716+0000","last_undegraded":"2024-09-22T02:59:46.124716+0000","last_fullsized":"2024-09-22T02:59:46.124716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1180,"log_dups_size":0,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77398016,"num_objects":1181,"num_object_clones":0,"num_object_copies":3543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1181,"num_write_kb":75584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":37,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-22T02:59:44.159004+0000","last_change":"2024-09-22T02:58:47.347756+0000","last_active":"2024-09-22T02:59:44.159004+0000","last_peered":"2024-09-22T02:59:44.159004+0000","last_clean":"2024-09-22T02:59:44.159004+0000","last_became_active":"2024-09-22T02:58:45.389306+0000","last_became_peered":"2024-09-22T02:58:45.389306+0000","last_unstale":"2024-09-22T02:59:44.159004+0000","last_undegraded":"2024-09-22T02:59:44.159004+0000","last_fullsized":"2024-09-22T02:59:44.159004+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1338441728,"num_objects":20423,"num_object_clones":0,"num_object_copies":61269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20423,"num_write_kb":1307072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":250888192,"data_stored":4014211072,"data_compressed":190612891,"data_compressed_allocated":250888192,"data_compressed_original":4014211072,"omap_allocated":0,"internal_metadata":0},"log_size":20407,"ondisk_log_size":20407,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":128432,"kb_used_data":71072,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94243408,"statfs":{"total":96636764160,"available":96505249792,"internally_reserved":0,"allocated":72777728,"data_stored":1161614759,"data_compressed":55156119,"data_compressed_allocated":72597504,"data_compressed_original":1161560064,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81740,"kb_used_data":54908,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94290100,"statfs":{"total":96636764160,"available":96553062400,"internally_reserved":0,"allocated":56225792,"data_stored":896391111,"data_compressed":42563997,"data_compressed_allocated":56025088,"data_compressed_original":896319488,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82808,"kb_used_data":55972,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94289032,"statfs":{"total":96636764160,"available":96551968768,"internally_reserved":0,"allocated":57315328,"data_stored":913889204,"data_compressed":43394922,"data_compressed_allocated":57118720,"data_compressed_original":913817600,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91064,"kb_used_data":60072,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94280776,"statfs":{"total":96636764160,"available":96543514624,"internally_reserved":0,"allocated":61513728,"data_stored":980998087,"data_compressed":46581538,"data_compressed_allocated":61313024,"data_compressed_original":980926464,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57942016,"data_stored":927072256,"data_compressed":44021595,"data_compressed_allocated":57942016,"data_compressed_original":927072256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62316544,"data_stored":997064704,"data_compressed":47345139,"data_compressed_allocated":62316544,"data_compressed_original":997064704,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":56913920,"data_stored":910622720,"data_compressed":43240462,"data_compressed_allocated":56913920,"data_compressed_original":910622720,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73715712,"data_stored":1179451392,"data_compressed":56005695,"data_compressed_allocated":73715712,"data_compressed_original":1179451392,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T02:59:51.755 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 55 16 22082 22066 25.073 21.25 0.0428755 0.0398566 2024-09-22T02:59:52.756 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 56 16 22435 22419 25.0192 22.0625 0.0430913 0.0399477 2024-09-22T02:59:53.652 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:53.756 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 57 16 22784 22768 24.9629 21.8125 0.0440652 0.0400397 2024-09-22T02:59:53.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:53.795+0000 7f8e50e23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:53.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:53.805+0000 7f8e50e23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:54.756 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 58 16 23134 23118 24.9096 21.875 0.0550821 0.0401214 2024-09-22T02:59:55.756 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 59 16 23481 23465 24.855 21.6875 0.0536496 0.0402122 2024-09-22T02:59:56.159 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T02:59:56.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:56.314+0000 7f53bb90e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:56.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:56.321+0000 7f53bb90e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:56.617 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T02:59:56.617 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T02:59:56.617 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3] 2024-09-22T02:59:56.618 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T02:59:56.618 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T02:59:56.756 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-22T02:59:56.755635+0000 min lat: 0.0084511 max lat: 0.122275 avg lat: 0.0402884 2024-09-22T02:59:56.756 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T02:59:56.756 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 60 16 23796 23780 24.7688 19.6875 0.0218198 0.0402884 2024-09-22T02:59:57.756 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 61 16 23796 23780 24.3627 0 - 0.0402884 2024-09-22T02:59:58.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T02:59:58.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:58.712+0000 7fbf56d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:58.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T02:59:58.716+0000 7fbf56d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T02:59:58.756 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 62 16 23796 23780 23.9697 0 - 0.0402884 2024-09-22T02:59:59.756 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 63 16 23796 23780 23.5893 0 - 0.0402884 2024-09-22T03:00:00.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:00:00.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 64 16 23796 23780 23.2207 0 - 0.0402884 2024-09-22T03:00:01.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 65 16 23796 23780 22.8634 0 - 0.0402884 2024-09-22T03:00:02.718 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:00:02.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 66 16 23796 23780 22.517 0 - 0.0402884 2024-09-22T03:00:03.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 67 16 23796 23780 22.1809 0 - 0.0402884 2024-09-22T03:00:04.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 68 16 23796 23780 21.8546 0 - 0.0402884 2024-09-22T03:00:05.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 69 16 23796 23780 21.5379 0 - 0.0402884 2024-09-22T03:00:06.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:00:06.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 70 16 23796 23780 21.2302 0 - 0.0402884 2024-09-22T03:00:07.719 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3] 2024-09-22T03:00:07.719 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:00:07.719 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-22T03:00:07.719 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:00:07.719 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:00:07.722 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:00:07.722 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:00:07.757 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 71 16 23796 23780 20.9312 0 - 0.0402884 2024-09-22T03:00:07.846 INFO:teuthology.orchestra.run.smithi081.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:00:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:00:07.850 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:00:08.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 72 16 23796 23780 20.6405 0 - 0.0402884 2024-09-22T03:00:09.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 73 16 23796 23780 20.3577 0 - 0.0402884 2024-09-22T03:00:10.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 74 16 23796 23780 20.0826 0 - 0.0402884 2024-09-22T03:00:11.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 75 16 23796 23780 19.8148 0 - 0.0402884 2024-09-22T03:00:12.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 76 16 23796 23780 19.5541 0 - 0.0402884 2024-09-22T03:00:12.851 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:00:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:00:12.951 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:00:12.951 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:00:13.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 77 16 23796 23780 19.3002 0 - 0.0402884 2024-09-22T03:00:14.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 78 16 23796 23780 19.0527 0 - 0.0402884 2024-09-22T03:00:15.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 79 16 23796 23780 18.8116 0 - 0.0402884 2024-09-22T03:00:16.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-22T03:00:16.758018+0000 min lat: 0.0084511 max lat: 0.122275 avg lat: 0.0402884 2024-09-22T03:00:16.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:00:16.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 80 16 23796 23780 18.5764 0 - 0.0402884 2024-09-22T03:00:17.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 81 16 23796 23780 18.3471 0 - 0.0402884 2024-09-22T03:00:17.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:00:18.047 INFO:teuthology.orchestra.run.smithi081.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:00:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:00:18.050 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:00:18.758 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 82 16 23796 23780 18.1233 0 - 0.0402884 2024-09-22T03:00:19.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 83 16 23796 23780 17.905 0 - 0.0402884 2024-09-22T03:00:20.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 84 16 23796 23780 17.6918 0 - 0.0402884 2024-09-22T03:00:21.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 85 16 23796 23780 17.4837 0 - 0.0402884 2024-09-22T03:00:22.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 86 16 23796 23780 17.2804 0 - 0.0402884 2024-09-22T03:00:23.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:00:23.147 INFO:teuthology.orchestra.run.smithi081.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:00:23.149 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:00:23.150 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:00:23.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 87 16 23796 23780 17.0818 0 - 0.0402884 2024-09-22T03:00:24.076 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:00:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:00:24.167 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:00:24.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 88 16 23796 23780 16.8876 0 - 0.0402884 2024-09-22T03:00:25.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 89 16 23796 23780 16.6979 0 - 0.0402884 2024-09-22T03:00:26.168 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:00:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:26.301+0000 7f9fec04f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:26.307+0000 7f9fec04f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:26.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 90 16 23936 23920 16.6096 0.291667 0.0494394 0.0601813 2024-09-22T03:00:27.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 91 16 24271 24255 16.6571 20.9375 0.0486568 0.0600124 2024-09-22T03:00:28.151 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:00:28.349 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-22T03:00:28.350 INFO:teuthology.orchestra.run.smithi081.stdout: "ops_in_flight": [ 2024-09-22T03:00:28.350 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.350 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.350 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 690, 2024-09-22T03:00:28.350 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.350 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.350 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.350 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.278493+0000" 2024-09-22T03:00:28.350 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.350 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.278503+0000" 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.278680+0000" 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.351 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.352 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.352 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.352 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.352 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.352 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.278791+0000" 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.353 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.279151+0000" 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.355 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.355 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.355 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.355 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.355 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.355 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.355 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.279700+0000" 2024-09-22T03:00:28.355 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.355 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.355 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.355 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.280194+0000" 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.356 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.357 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.357 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.357 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.357 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.357 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.357 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.357 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.357 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.357 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.357 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.357 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.358 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.358 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.358 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.280400+0000" 2024-09-22T03:00:28.358 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.358 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.358 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.358 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.358 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.358 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.359 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.280815+0000" 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.360 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.361 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.290238+0000" 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.362 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 694, 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.294064+0000" 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.294077+0000" 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.296462+0000" 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.365 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.366 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.366 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.366 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.366 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.366 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.366 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.366 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.366 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.366 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.366 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.297316+0000" 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.297728+0000" 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.367 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.368 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.368 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.368 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.368 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.368 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.368 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.368 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.368 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.368 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.368 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.299415+0000" 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.369 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.300763+0000" 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.301726+0000" 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.372 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.373 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.373 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.373 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.373 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.373 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.373 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.373 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.373 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.304157+0000" 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.374 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.306712+0000" 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.375 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "replicated_request", 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 695, 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: "RepRequest": { 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: "reqid": "client.4243.0:24436", 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "3.0", 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: "map_epoch": 24, 2024-09-22T03:00:28.377 INFO:teuthology.orchestra.run.smithi081.stdout: "min_epoch": 23, 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: "oid": "3:08815bfc:::benchmark_data_smithi081_30037_object24435:head", 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: "from": "3" 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 595, 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.378 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.886654+0000" 2024-09-22T03:00:28.379 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.379 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.379 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.379 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.886660+0000" 2024-09-22T03:00:28.379 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.379 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.379 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.379 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.379 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.379 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.886759+0000" 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.380 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.886849+0000" 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.886941+0000" 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.926214+0000" 2024-09-22T03:00:28.382 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.887259+0000" 2024-09-22T03:00:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.383 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.887399+0000" 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.384 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.887565+0000" 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.385 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.386 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.887785+0000" 2024-09-22T03:00:28.387 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.388 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.889062+0000" 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.924790+0000" 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.390 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 601, 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.907570+0000" 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.907576+0000" 2024-09-22T03:00:28.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.907678+0000" 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.392 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.393 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.393 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.393 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.393 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.393 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.393 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.393 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.907760+0000" 2024-09-22T03:00:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.395 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.907830+0000" 2024-09-22T03:00:28.395 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.395 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.395 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.395 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.395 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.395 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.395 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.395 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.396 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.396 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.396 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.396 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.933005+0000" 2024-09-22T03:00:28.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.397 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.397 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.397 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.908045+0000" 2024-09-22T03:00:28.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.397 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.397 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.397 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.397 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.398 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.398 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.398 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.398 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.398 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.398 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.398 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.399 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.908153+0000" 2024-09-22T03:00:28.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.399 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.399 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.399 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.399 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.399 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.399 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.908241+0000" 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.400 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.401 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.908487+0000" 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.910285+0000" 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.932030+0000" 2024-09-22T03:00:28.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 602, 2024-09-22T03:00:28.405 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.910864+0000" 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.910870+0000" 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.406 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.914228+0000" 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.915125+0000" 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.915433+0000" 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.942941+0000" 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.409 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.916936+0000" 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.917131+0000" 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.411 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.917243+0000" 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.413 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.918312+0000" 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.919399+0000" 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.942393+0000" 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 604, 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.918926+0000" 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.918932+0000" 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.919038+0000" 2024-09-22T03:00:28.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.419 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.919166+0000" 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.919451+0000" 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.420 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.959157+0000" 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.421 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.919931+0000" 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.422 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.923142+0000" 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.423 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.923412+0000" 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.923888+0000" 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.426 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.927660+0000" 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.958255+0000" 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 606, 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.933731+0000" 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.933736+0000" 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.430 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.933814+0000" 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.431 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.933862+0000" 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.933930+0000" 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.432 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.980139+0000" 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.433 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.934125+0000" 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.934221+0000" 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.934314+0000" 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.437 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.934457+0000" 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.438 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.935332+0000" 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.439 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.978405+0000" 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.440 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 607, 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.441 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.943404+0000" 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.943409+0000" 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.943482+0000" 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.442 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.443 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.943779+0000" 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.944033+0000" 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.444 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.987331+0000" 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.944238+0000" 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.446 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.944353+0000" 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.944447+0000" 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.448 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.944721+0000" 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.946074+0000" 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.985766+0000" 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.452 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 613, 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.453 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.960164+0000" 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.960170+0000" 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.960473+0000" 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.454 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.455 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.960733+0000" 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.960849+0000" 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.456 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.017465+0000" 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.457 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.961205+0000" 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.458 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.961315+0000" 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.961423+0000" 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.460 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.461 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.962161+0000" 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.462 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.963980+0000" 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.463 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.015443+0000" 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.464 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 616, 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.971647+0000" 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.465 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.971662+0000" 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.972077+0000" 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.972411+0000" 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.972542+0000" 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.025940+0000" 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.973264+0000" 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.973698+0000" 2024-09-22T03:00:28.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.471 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.973920+0000" 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.473 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.974139+0000" 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.474 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.981052+0000" 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.475 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.024744+0000" 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 619, 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.988589+0000" 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.477 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.988595+0000" 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.988834+0000" 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.478 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.989025+0000" 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.989157+0000" 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.480 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.035274+0000" 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.989561+0000" 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.481 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.989812+0000" 2024-09-22T03:00:28.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.483 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.990229+0000" 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.484 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.485 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.993556+0000" 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.996096+0000" 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.487 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.034462+0000" 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 625, 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.005647+0000" 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.005656+0000" 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.006087+0000" 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.490 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.007382+0000" 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.491 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.007486+0000" 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.492 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.054720+0000" 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.008407+0000" 2024-09-22T03:00:28.493 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.494 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.008578+0000" 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.495 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.008800+0000" 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.496 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.497 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.009053+0000" 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.498 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.010756+0000" 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.499 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.052632+0000" 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.500 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 627, 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.016395+0000" 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.501 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.016402+0000" 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.017014+0000" 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.502 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.017127+0000" 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.503 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.017762+0000" 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.504 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.067299+0000" 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.018115+0000" 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.505 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.018743+0000" 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.506 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.507 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.018878+0000" 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.508 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.509 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.019408+0000" 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.510 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.020721+0000" 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.511 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.062068+0000" 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.512 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 631, 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.030763+0000" 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.030773+0000" 2024-09-22T03:00:28.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.031049+0000" 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.514 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.031264+0000" 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.031369+0000" 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.516 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.084249+0000" 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.031705+0000" 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.031968+0000" 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.518 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.032128+0000" 2024-09-22T03:00:28.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.520 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.521 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.032802+0000" 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.522 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.034269+0000" 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.523 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.082285+0000" 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 634, 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.041732+0000" 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.041740+0000" 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.042044+0000" 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.526 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.527 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.527 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.527 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.527 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.527 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.527 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.527 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.527 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.527 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.527 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.042266+0000" 2024-09-22T03:00:28.527 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.528 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.528 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.528 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.528 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.528 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.528 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.528 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.042331+0000" 2024-09-22T03:00:28.528 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.529 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.086996+0000" 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.042582+0000" 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.042723+0000" 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.531 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.042858+0000" 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.532 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.043279+0000" 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.046109+0000" 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.535 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.536 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.086482+0000" 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.537 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 636, 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.056250+0000" 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.056255+0000" 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.538 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.056379+0000" 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.539 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.540 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.540 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.540 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.056457+0000" 2024-09-22T03:00:28.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.540 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.541 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.541 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.541 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.541 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.056591+0000" 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.542 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.092728+0000" 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.056804+0000" 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.057078+0000" 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.545 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.057320+0000" 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.546 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.547 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.057725+0000" 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.548 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.062049+0000" 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.549 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.550 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.091556+0000" 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.551 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 640, 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.073066+0000" 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.073076+0000" 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.552 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.073389+0000" 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.553 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.073661+0000" 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.074918+0000" 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.554 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.097402+0000" 2024-09-22T03:00:28.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.075131+0000" 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.556 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.075250+0000" 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.557 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.075472+0000" 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.559 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.075797+0000" 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.560 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.077493+0000" 2024-09-22T03:00:28.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.096276+0000" 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.563 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 651, 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.101073+0000" 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.101081+0000" 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.564 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.103881+0000" 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.565 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.104397+0000" 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.566 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.104590+0000" 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.142151+0000" 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.104846+0000" 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.568 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.106654+0000" 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.569 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.107789+0000" 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.570 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.571 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.572 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.108789+0000" 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.114621+0000" 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.574 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.141811+0000" 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.575 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 666, 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.162501+0000" 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.162515+0000" 2024-09-22T03:00:28.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.162662+0000" 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.577 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.162746+0000" 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.162839+0000" 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.579 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.213772+0000" 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.163489+0000" 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.580 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.163620+0000" 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.581 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.163751+0000" 2024-09-22T03:00:28.582 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.583 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.584 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.163898+0000" 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.585 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.166585+0000" 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.586 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.213149+0000" 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.587 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 670, 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.588 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.181840+0000" 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.181849+0000" 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.182374+0000" 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.590 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.182441+0000" 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.182493+0000" 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.591 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.223612+0000" 2024-09-22T03:00:28.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.182754+0000" 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.593 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.182856+0000" 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.594 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.182952+0000" 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.597 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.183088+0000" 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.598 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.184340+0000" 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.218196+0000" 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.600 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 671, 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.189426+0000" 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.189433+0000" 2024-09-22T03:00:28.601 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.189552+0000" 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.602 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.189654+0000" 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.190482+0000" 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.604 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.254101+0000" 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.190728+0000" 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.190836+0000" 2024-09-22T03:00:28.606 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.607 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.190944+0000" 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.608 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.609 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.191081+0000" 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.192158+0000" 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.252711+0000" 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 495, 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.459494+0000" 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.459508+0000" 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.461124+0000" 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.614 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.461244+0000" 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.615 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.461658+0000" 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.495296+0000" 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.462017+0000" 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.617 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.618 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.462184+0000" 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.619 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.462357+0000" 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.620 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.462635+0000" 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.622 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.463604+0000" 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.493983+0000" 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.624 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 502, 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.479425+0000" 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.625 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.479433+0000" 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.480375+0000" 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.480966+0000" 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.481238+0000" 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.530042+0000" 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.481887+0000" 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.484453+0000" 2024-09-22T03:00:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.487556+0000" 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.487799+0000" 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.488944+0000" 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.529485+0000" 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 506, 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.491133+0000" 2024-09-22T03:00:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.491140+0000" 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.491216+0000" 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.639 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.639 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.639 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.639 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.639 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.639 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.639 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.491263+0000" 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.493638+0000" 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.640 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.540787+0000" 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.641 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.494284+0000" 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.642 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.494468+0000" 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.643 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.494810+0000" 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.645 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.495085+0000" 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.500207+0000" 2024-09-22T03:00:28.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.648 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.540358+0000" 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.649 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 518, 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.546488+0000" 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.546498+0000" 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.650 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.546620+0000" 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.651 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.547132+0000" 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.547495+0000" 2024-09-22T03:00:28.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.653 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.596674+0000" 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.547855+0000" 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.654 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.548032+0000" 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.655 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.548187+0000" 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.656 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.548450+0000" 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.658 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.659 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.549482+0000" 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.660 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.596427+0000" 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 532, 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.595020+0000" 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.595025+0000" 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.662 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.595368+0000" 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.595721+0000" 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.664 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.598963+0000" 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.635476+0000" 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.599190+0000" 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.666 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.599299+0000" 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.667 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.599468+0000" 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.668 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.669 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.670 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.599738+0000" 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.601091+0000" 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.672 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.635044+0000" 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.673 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 537, 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.609210+0000" 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.609215+0000" 2024-09-22T03:00:28.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.612069+0000" 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.675 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.612244+0000" 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.676 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.613593+0000" 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.641879+0000" 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.613996+0000" 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.679 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.614177+0000" 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.680 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.614335+0000" 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.614568+0000" 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.683 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.615427+0000" 2024-09-22T03:00:28.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.685 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.641388+0000" 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 538, 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.620955+0000" 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.620963+0000" 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.621141+0000" 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.688 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.621987+0000" 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.622633+0000" 2024-09-22T03:00:28.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.690 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.659566+0000" 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.623019+0000" 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.691 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.623216+0000" 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.692 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.623453+0000" 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.694 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.695 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.623695+0000" 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.624764+0000" 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.697 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.659131+0000" 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 542, 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.625184+0000" 2024-09-22T03:00:28.699 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.625188+0000" 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.625336+0000" 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.700 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.625409+0000" 2024-09-22T03:00:28.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.629438+0000" 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.702 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.672543+0000" 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.703 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.629884+0000" 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.704 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.630243+0000" 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.630410+0000" 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.706 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.707 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.634392+0000" 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.708 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.636113+0000" 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.709 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.672091+0000" 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 567, 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.711 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.751294+0000" 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.751304+0000" 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.751640+0000" 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.713 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.751858+0000" 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.752926+0000" 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.714 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.783291+0000" 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.715 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.753240+0000" 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.716 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.753434+0000" 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.717 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.753616+0000" 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.719 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.753889+0000" 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.720 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.758047+0000" 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.721 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.782832+0000" 2024-09-22T03:00:28.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 576, 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.723 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.796723+0000" 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.796731+0000" 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.797049+0000" 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.724 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.797155+0000" 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.725 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.800967+0000" 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.823617+0000" 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.801331+0000" 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.727 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.801530+0000" 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.801685+0000" 2024-09-22T03:00:28.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.730 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.731 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.801937+0000" 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.802845+0000" 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.823187+0000" 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 626, 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.009837+0000" 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.735 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.009843+0000" 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.009950+0000" 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.736 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.010059+0000" 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.011470+0000" 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.058144+0000" 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.011846+0000" 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.739 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.012033+0000" 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.740 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.012203+0000" 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.742 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.744 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.744 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.744 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.012438+0000" 2024-09-22T03:00:28.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.745 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.745 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.013383+0000" 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.747 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.057687+0000" 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.748 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 629, 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.025804+0000" 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.025815+0000" 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.749 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.027776+0000" 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.750 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.028063+0000" 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.029405+0000" 2024-09-22T03:00:28.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.752 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.087236+0000" 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.029789+0000" 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.753 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.029984+0000" 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.754 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.030155+0000" 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.755 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.756 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.030404+0000" 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.757 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.038604+0000" 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.758 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.086451+0000" 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 643, 2024-09-22T03:00:28.760 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.082223+0000" 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.082233+0000" 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.761 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.083569+0000" 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.762 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.083744+0000" 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.084674+0000" 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.125678+0000" 2024-09-22T03:00:28.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.085036+0000" 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.765 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.085213+0000" 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.766 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.085397+0000" 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.085684+0000" 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.087083+0000" 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.770 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.771 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.125154+0000" 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 648, 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.095424+0000" 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.095437+0000" 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.773 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.095892+0000" 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.774 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.096015+0000" 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.096375+0000" 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.137247+0000" 2024-09-22T03:00:28.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.096721+0000" 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.096886+0000" 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.778 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.097063+0000" 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.780 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.781 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.097293+0000" 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.098271+0000" 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.132629+0000" 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.784 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 649, 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.098323+0000" 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.785 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.098334+0000" 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.098458+0000" 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.786 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.098577+0000" 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.102351+0000" 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.788 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.143653+0000" 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.789 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.102768+0000" 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.102952+0000" 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.103125+0000" 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.792 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.103353+0000" 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.794 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.795 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.104598+0000" 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.796 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.140842+0000" 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 657, 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.128399+0000" 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.128409+0000" 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.798 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.128552+0000" 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.129140+0000" 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.800 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.132221+0000" 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.209885+0000" 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.133240+0000" 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.802 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.137416+0000" 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.138198+0000" 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.804 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.806 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.138660+0000" 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.141732+0000" 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.209437+0000" 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.809 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 658, 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.130257+0000" 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.810 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.130262+0000" 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.130432+0000" 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.130580+0000" 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.812 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.132590+0000" 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.217558+0000" 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.138079+0000" 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.815 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.138379+0000" 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.138875+0000" 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.818 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.139516+0000" 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.141972+0000" 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.142942+0000" 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.820 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.217084+0000" 2024-09-22T03:00:28.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 661, 2024-09-22T03:00:28.822 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.137959+0000" 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.137971+0000" 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.823 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.138153+0000" 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.824 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.138674+0000" 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.139998+0000" 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.236230+0000" 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.826 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.140403+0000" 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.140808+0000" 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.828 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.141352+0000" 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.142123+0000" 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.148163+0000" 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.148832+0000" 2024-09-22T03:00:28.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.833 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.235750+0000" 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.834 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 682, 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.235908+0000" 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.235918+0000" 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.835 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.236121+0000" 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.236242+0000" 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.237557+0000" 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.837 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.276888+0000" 2024-09-22T03:00:28.838 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.237923+0000" 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.839 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.238097+0000" 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.840 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.238271+0000" 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.842 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.238519+0000" 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.843 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.241420+0000" 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.844 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.276448+0000" 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "replicated_request", 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 697, 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.846 INFO:teuthology.orchestra.run.smithi081.stdout: "RepRequest": { 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: "reqid": "client.4243.0:24443", 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "3.6", 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: "map_epoch": 24, 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: "min_epoch": 23, 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: "oid": "3:60a5aa6a:::benchmark_data_smithi081_30037_object24442:head", 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: "from": "2" 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "replicated_request", 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 699, 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: "RepRequest": { 2024-09-22T03:00:28.847 INFO:teuthology.orchestra.run.smithi081.stdout: "reqid": "client.4243.0:24445", 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "3.6", 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: "map_epoch": 24, 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: "min_epoch": 23, 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: "oid": "3:68d69619:::benchmark_data_smithi081_30037_object24444:head", 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: "from": "2" 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 435, 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.848 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.207700+0000" 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.207707+0000" 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.208418+0000" 2024-09-22T03:00:28.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.850 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.208564+0000" 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.209486+0000" 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.851 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.274545+0000" 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.852 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.209689+0000" 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.209780+0000" 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.854 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.209885+0000" 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.855 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.856 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.210039+0000" 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.210785+0000" 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.858 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.274264+0000" 2024-09-22T03:00:28.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 455, 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.284301+0000" 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.284307+0000" 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.861 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.284374+0000" 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.284423+0000" 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.287317+0000" 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.863 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.864 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.335944+0000" 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.287680+0000" 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.287862+0000" 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.866 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.288031+0000" 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.867 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.288278+0000" 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.869 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.289313+0000" 2024-09-22T03:00:28.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.871 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.332548+0000" 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.872 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 470, 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.344135+0000" 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.344142+0000" 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.344217+0000" 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.874 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.344262+0000" 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.344417+0000" 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.876 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.369899+0000" 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.344623+0000" 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.344737+0000" 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.878 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.344832+0000" 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.879 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.880 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.881 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.344987+0000" 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.345726+0000" 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.883 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.369657+0000" 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.884 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 477, 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.373159+0000" 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.885 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.373163+0000" 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.373236+0000" 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.373290+0000" 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.373439+0000" 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.397421+0000" 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.373666+0000" 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.889 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.373758+0000" 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.890 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.373846+0000" 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.892 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.373993+0000" 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.894 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.374553+0000" 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.397023+0000" 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.896 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 484, 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.410849+0000" 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.897 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.410856+0000" 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.411177+0000" 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.898 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.411229+0000" 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.899 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.412237+0000" 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.900 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.446077+0000" 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.412560+0000" 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.901 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.902 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.415366+0000" 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.415458+0000" 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.904 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.905 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.415599+0000" 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.906 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.416221+0000" 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.907 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.445646+0000" 2024-09-22T03:00:28.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 497, 2024-09-22T03:00:28.909 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.464720+0000" 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.464726+0000" 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.464924+0000" 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.911 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.465042+0000" 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.912 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.465346+0000" 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.913 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.493428+0000" 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.465557+0000" 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.914 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.915 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.915 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.915 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.915 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.915 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.915 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.915 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.915 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.465662+0000" 2024-09-22T03:00:28.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.916 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.465759+0000" 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.917 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.918 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.465900+0000" 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.919 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.466534+0000" 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.493183+0000" 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.921 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 508, 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.501519+0000" 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.922 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.501527+0000" 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.502208+0000" 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.502312+0000" 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.924 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.502750+0000" 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.546968+0000" 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.503105+0000" 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.926 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.503280+0000" 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.928 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.503437+0000" 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.929 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.930 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.503681+0000" 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.931 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.507937+0000" 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.932 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.546507+0000" 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.933 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 510, 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.503219+0000" 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.934 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.503224+0000" 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.503283+0000" 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.935 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.936 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.936 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.936 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.936 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.936 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.936 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.936 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.936 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.936 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.936 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.503359+0000" 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.506750+0000" 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.937 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.549736+0000" 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.938 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.506943+0000" 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.939 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.507037+0000" 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.940 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.507129+0000" 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.941 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.942 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.507264+0000" 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.943 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.512680+0000" 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.944 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.549330+0000" 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.945 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 528, 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.590376+0000" 2024-09-22T03:00:28.946 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.590385+0000" 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.590592+0000" 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.947 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.949 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.949 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.949 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.949 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.949 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.591025+0000" 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.950 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.592911+0000" 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.951 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.627953+0000" 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.593244+0000" 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.952 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.593405+0000" 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.953 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.593569+0000" 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.954 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.955 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.593796+0000" 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.956 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.594693+0000" 2024-09-22T03:00:28.957 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.627698+0000" 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 546, 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.641863+0000" 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.641870+0000" 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.642490+0000" 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.961 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.642619+0000" 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.643074+0000" 2024-09-22T03:00:28.962 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.963 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.678050+0000" 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.643427+0000" 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.964 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.643610+0000" 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.965 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.643785+0000" 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.966 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.967 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.643932+0000" 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.968 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.644588+0000" 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.677614+0000" 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.970 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 557, 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.971 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.715890+0000" 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.715896+0000" 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.716013+0000" 2024-09-22T03:00:28.972 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.973 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.716111+0000" 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.716711+0000" 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.974 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.763525+0000" 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.975 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.716937+0000" 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.717057+0000" 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.977 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.717220+0000" 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.979 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.717481+0000" 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.718165+0000" 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.981 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.982 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.763077+0000" 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 558, 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.983 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.717563+0000" 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.717569+0000" 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.984 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.717696+0000" 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.985 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.718539+0000" 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.722496+0000" 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.986 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.774600+0000" 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.722959+0000" 2024-09-22T03:00:28.987 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.988 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.723133+0000" 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.989 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.726995+0000" 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.990 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:28.991 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.727286+0000" 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.992 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.732402+0000" 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.774347+0000" 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:28.995 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 565, 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.747504+0000" 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.747515+0000" 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.747608+0000" 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.997 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.748117+0000" 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.748337+0000" 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.998 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.782829+0000" 2024-09-22T03:00:28.999 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.748559+0000" 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.000 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.748671+0000" 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.001 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.748767+0000" 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:29.002 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.003 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.748924+0000" 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.749604+0000" 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:29.006 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.782540+0000" 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 581, 2024-09-22T03:00:29.007 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.813577+0000" 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.813587+0000" 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.008 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.813686+0000" 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.009 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.813741+0000" 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.813897+0000" 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.010 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.827518+0000" 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.011 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.814249+0000" 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.012 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.814422+0000" 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.013 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.814599+0000" 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.014 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:29.015 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.814844+0000" 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.016 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.815777+0000" 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.017 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.827065+0000" 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.018 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 622, 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.994552+0000" 2024-09-22T03:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.994564+0000" 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.994813+0000" 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.020 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.995680+0000" 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.021 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.997561+0000" 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:29.022 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.043937+0000" 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.997775+0000" 2024-09-22T03:00:29.023 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.024 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.997880+0000" 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.025 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.997986+0000" 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.026 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.027 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.998124+0000" 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.028 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:27.998743+0000" 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.029 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.043500+0000" 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.030 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 628, 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.031 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.019857+0000" 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.019862+0000" 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.020167+0000" 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.032 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.020238+0000" 2024-09-22T03:00:29.033 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.020831+0000" 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.034 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.083967+0000" 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.021057+0000" 2024-09-22T03:00:29.035 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.036 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.021156+0000" 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.037 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.021250+0000" 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.038 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.039 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.021388+0000" 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.040 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.021957+0000" 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.041 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.079982+0000" 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.042 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 641, 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.043 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.076506+0000" 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.076511+0000" 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.076653+0000" 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.044 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:29.045 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.076768+0000" 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.079050+0000" 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.046 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.127811+0000" 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.079410+0000" 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.047 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:29.048 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.079847+0000" 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.049 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.083356+0000" 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.050 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.051 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.083614+0000" 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:29.052 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.084667+0000" 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.053 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.125226+0000" 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 673, 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:29.055 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.196770+0000" 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.196782+0000" 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.197157+0000" 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.056 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.197516+0000" 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.057 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.203008+0000" 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.058 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.261917+0000" 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.203349+0000" 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.059 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.060 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.203526+0000" 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.061 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.203681+0000" 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:29.063 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.203933+0000" 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.064 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.204899+0000" 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.065 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.261634+0000" 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client_request", 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 679, 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.067 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.216368+0000" 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.216378+0000" 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.216600+0000" 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.068 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.216817+0000" 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.217461+0000" 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:00:29.070 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.271612+0000" 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.217813+0000" 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.071 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.217987+0000" 2024-09-22T03:00:29.072 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.073 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.218134+0000" 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.074 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.218342+0000" 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:00:29.076 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.220069+0000" 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.077 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: "blocking_event": { 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-09-22T03:00:28.271189+0000" 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:00:29.078 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.079 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:00:29.079 INFO:teuthology.orchestra.run.smithi081.stdout: "time_event": { 2024-09-22T03:00:29.079 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:00:29.079 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "0.000000" 2024-09-22T03:00:29.079 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.079 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.079 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:00:29.079 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T03:00:29.079 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-22T03:00:29.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:00:29.081 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 92 16 24604 24588 16.7023 20.8125 0.0449807 0.0598485 2024-09-22T03:00:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:29.267+0000 7fde20797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:29.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:29.274+0000 7fde20797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:29.759 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 93 16 24924 24908 16.7377 20 0.0404063 0.0597243 2024-09-22T03:00:30.760 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 94 16 25269 25253 16.789 21.5625 0.0466957 0.0595406 2024-09-22T03:00:31.547 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:00:31.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:31.686+0000 7f8acfd57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:31.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:31.693+0000 7f8acfd57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:31.760 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 95 16 25589 25573 16.8228 20 0.0232856 0.0594111 2024-09-22T03:00:32.760 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 96 16 25916 25900 16.8604 20.4375 0.0273816 0.0592901 2024-09-22T03:00:33.760 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 97 16 26252 26236 16.9031 21 0.0306694 0.0591444 2024-09-22T03:00:33.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:00:34.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:34.050+0000 7f3807e18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:34.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:34.057+0000 7f3807e18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:34.081 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-09-22T03:00:34.081 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:00:34.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:00:34.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:34.223+0000 7f04c3c18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:34.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:34.231+0000 7f04c3c18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:34.561 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:00:34.561 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:00:34.580 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-09-22T03:00:34.306145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1681392179,"num_objects":25651,"num_object_clones":0,"num_object_copies":76953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":25786,"num_write_kb":1643304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25731,"ondisk_log_size":25731,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":499300,"kb_used_data":305440,"kb_used_omap":0,"kb_used_meta":193792,"kb_avail":376988060,"statfs":{"total":386547056640,"available":386035773440,"internally_reserved":0,"allocated":312770560,"data_stored":4990371621,"data_compressed":236959533,"data_compressed_allocated":311894016,"data_compressed_original":4990058496,"omap_allocated":0,"internal_metadata":198443008},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":123273216,"num_objects":1881,"num_object_clones":0,"num_object_copies":5643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1881,"num_write_kb":120384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.215956"},"pg_stats":[{"pgid":"3.f","version":"24'1626","reported_seq":1668,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.316131+0000","last_change":"2024-09-22T03:00:26.313376+0000","last_active":"2024-09-22T03:00:32.316131+0000","last_peered":"2024-09-22T03:00:32.316131+0000","last_clean":"2024-09-22T03:00:32.316131+0000","last_became_active":"2024-09-22T03:00:26.186244+0000","last_became_peered":"2024-09-22T03:00:26.186244+0000","last_unstale":"2024-09-22T03:00:32.316131+0000","last_undegraded":"2024-09-22T03:00:32.316131+0000","last_fullsized":"2024-09-22T03:00:32.316131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1626,"log_dups_size":0,"ondisk_log_size":1626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106627072,"num_objects":1627,"num_object_clones":0,"num_object_copies":4881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1627,"num_write_kb":104128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"24'1592","reported_seq":1632,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.261200+0000","last_change":"2024-09-22T03:00:26.266675+0000","last_active":"2024-09-22T03:00:32.261200+0000","last_peered":"2024-09-22T03:00:32.261200+0000","last_clean":"2024-09-22T03:00:32.261200+0000","last_became_active":"2024-09-22T03:00:26.190579+0000","last_became_peered":"2024-09-22T03:00:26.190579+0000","last_unstale":"2024-09-22T03:00:32.261200+0000","last_undegraded":"2024-09-22T03:00:32.261200+0000","last_fullsized":"2024-09-22T03:00:32.261200+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1592,"log_dups_size":0,"ondisk_log_size":1592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104398848,"num_objects":1593,"num_object_clones":0,"num_object_copies":4779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1593,"num_write_kb":101952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1595","reported_seq":1629,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:30.822180+0000","last_change":"2024-09-22T03:00:26.238897+0000","last_active":"2024-09-22T03:00:30.822180+0000","last_peered":"2024-09-22T03:00:30.822180+0000","last_clean":"2024-09-22T03:00:30.822180+0000","last_became_active":"2024-09-22T03:00:26.234315+0000","last_became_peered":"2024-09-22T03:00:26.234315+0000","last_unstale":"2024-09-22T03:00:30.822180+0000","last_undegraded":"2024-09-22T03:00:30.822180+0000","last_fullsized":"2024-09-22T03:00:30.822180+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1595,"log_dups_size":0,"ondisk_log_size":1595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104595456,"num_objects":1596,"num_object_clones":0,"num_object_copies":4788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1596,"num_write_kb":102144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"24'1556","reported_seq":1590,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.312896+0000","last_change":"2024-09-22T02:58:55.740009+0000","last_active":"2024-09-22T03:00:32.312896+0000","last_peered":"2024-09-22T03:00:32.312896+0000","last_clean":"2024-09-22T03:00:32.312896+0000","last_became_active":"2024-09-22T02:58:55.731779+0000","last_became_peered":"2024-09-22T02:58:55.731779+0000","last_unstale":"2024-09-22T03:00:32.312896+0000","last_undegraded":"2024-09-22T03:00:32.312896+0000","last_fullsized":"2024-09-22T03:00:32.312896+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1556,"log_dups_size":0,"ondisk_log_size":1556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102039552,"num_objects":1557,"num_object_clones":0,"num_object_copies":4671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1557,"num_write_kb":99648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"24'1589","reported_seq":1623,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.238583+0000","last_change":"2024-09-22T02:58:55.742415+0000","last_active":"2024-09-22T03:00:32.238583+0000","last_peered":"2024-09-22T03:00:32.238583+0000","last_clean":"2024-09-22T03:00:32.238583+0000","last_became_active":"2024-09-22T02:58:55.732221+0000","last_became_peered":"2024-09-22T02:58:55.732221+0000","last_unstale":"2024-09-22T03:00:32.238583+0000","last_undegraded":"2024-09-22T03:00:32.238583+0000","last_fullsized":"2024-09-22T03:00:32.238583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1589,"log_dups_size":0,"ondisk_log_size":1589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104202240,"num_objects":1590,"num_object_clones":0,"num_object_copies":4770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1590,"num_write_kb":101760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"24'1573","reported_seq":1615,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.218989+0000","last_change":"2024-09-22T03:00:26.309582+0000","last_active":"2024-09-22T03:00:32.218989+0000","last_peered":"2024-09-22T03:00:32.218989+0000","last_clean":"2024-09-22T03:00:32.218989+0000","last_became_active":"2024-09-22T03:00:26.170722+0000","last_became_peered":"2024-09-22T03:00:26.170722+0000","last_unstale":"2024-09-22T03:00:32.218989+0000","last_undegraded":"2024-09-22T03:00:32.218989+0000","last_fullsized":"2024-09-22T03:00:32.218989+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103153664,"num_objects":1574,"num_object_clones":0,"num_object_copies":4722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1574,"num_write_kb":100736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"24'1584","reported_seq":1619,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.318478+0000","last_change":"2024-09-22T03:00:26.168672+0000","last_active":"2024-09-22T03:00:32.318478+0000","last_peered":"2024-09-22T03:00:32.318478+0000","last_clean":"2024-09-22T03:00:32.318478+0000","last_became_active":"2024-09-22T03:00:26.163224+0000","last_became_peered":"2024-09-22T03:00:26.163224+0000","last_unstale":"2024-09-22T03:00:32.318478+0000","last_undegraded":"2024-09-22T03:00:32.318478+0000","last_fullsized":"2024-09-22T03:00:32.318478+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103874560,"num_objects":1585,"num_object_clones":0,"num_object_copies":4755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1585,"num_write_kb":101440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"24'1631","reported_seq":1665,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:30.854760+0000","last_change":"2024-09-22T03:00:26.246290+0000","last_active":"2024-09-22T03:00:30.854760+0000","last_peered":"2024-09-22T03:00:30.854760+0000","last_clean":"2024-09-22T03:00:30.854760+0000","last_became_active":"2024-09-22T03:00:26.240145+0000","last_became_peered":"2024-09-22T03:00:26.240145+0000","last_unstale":"2024-09-22T03:00:30.854760+0000","last_undegraded":"2024-09-22T03:00:30.854760+0000","last_fullsized":"2024-09-22T03:00:30.854760+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1631,"log_dups_size":0,"ondisk_log_size":1631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106954752,"num_objects":1632,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1632,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:26.101419+0000","last_change":"2024-09-22T02:58:47.345653+0000","last_active":"2024-09-22T03:00:26.101419+0000","last_peered":"2024-09-22T03:00:26.101419+0000","last_clean":"2024-09-22T03:00:26.101419+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T03:00:26.101419+0000","last_undegraded":"2024-09-22T03:00:26.101419+0000","last_fullsized":"2024-09-22T03:00:26.101419+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"24'1588","reported_seq":1622,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:30.785649+0000","last_change":"2024-09-22T03:00:26.214675+0000","last_active":"2024-09-22T03:00:30.785649+0000","last_peered":"2024-09-22T03:00:30.785649+0000","last_clean":"2024-09-22T03:00:30.785649+0000","last_became_active":"2024-09-22T03:00:26.209674+0000","last_became_peered":"2024-09-22T03:00:26.209674+0000","last_unstale":"2024-09-22T03:00:30.785649+0000","last_undegraded":"2024-09-22T03:00:30.785649+0000","last_fullsized":"2024-09-22T03:00:30.785649+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1588,"log_dups_size":0,"ondisk_log_size":1588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104136704,"num_objects":1589,"num_object_clones":0,"num_object_copies":4767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1589,"num_write_kb":101696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:26.159050+0000","last_change":"2024-09-22T03:00:26.159687+0000","last_active":"2024-09-22T03:00:26.159050+0000","last_peered":"2024-09-22T03:00:26.159050+0000","last_clean":"2024-09-22T02:59:44.151322+0000","last_became_active":"2024-09-22T03:00:26.155775+0000","last_became_peered":"2024-09-22T03:00:26.155775+0000","last_unstale":"2024-09-22T03:00:26.159050+0000","last_undegraded":"2024-09-22T03:00:26.159050+0000","last_fullsized":"2024-09-22T03:00:26.159050+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"24'1644","reported_seq":1678,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.254600+0000","last_change":"2024-09-22T02:58:55.741407+0000","last_active":"2024-09-22T03:00:32.254600+0000","last_peered":"2024-09-22T03:00:32.254600+0000","last_clean":"2024-09-22T03:00:32.254600+0000","last_became_active":"2024-09-22T02:58:55.737022+0000","last_became_peered":"2024-09-22T02:58:55.737022+0000","last_unstale":"2024-09-22T03:00:32.254600+0000","last_undegraded":"2024-09-22T03:00:32.254600+0000","last_fullsized":"2024-09-22T03:00:32.254600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1644,"log_dups_size":0,"ondisk_log_size":1644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107806720,"num_objects":1645,"num_object_clones":0,"num_object_copies":4935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1645,"num_write_kb":105280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":49,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:26.169994+0000","last_change":"2024-09-22T03:00:26.171111+0000","last_active":"2024-09-22T03:00:26.169994+0000","last_peered":"2024-09-22T03:00:26.169994+0000","last_clean":"2024-09-22T02:59:44.139369+0000","last_became_active":"2024-09-22T03:00:26.162692+0000","last_became_peered":"2024-09-22T03:00:26.162692+0000","last_unstale":"2024-09-22T03:00:26.169994+0000","last_undegraded":"2024-09-22T03:00:26.169994+0000","last_fullsized":"2024-09-22T03:00:26.169994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"24'1571","reported_seq":1611,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.272093+0000","last_change":"2024-09-22T03:00:26.259241+0000","last_active":"2024-09-22T03:00:32.272093+0000","last_peered":"2024-09-22T03:00:32.272093+0000","last_clean":"2024-09-22T03:00:32.272093+0000","last_became_active":"2024-09-22T03:00:26.163223+0000","last_became_peered":"2024-09-22T03:00:26.163223+0000","last_unstale":"2024-09-22T03:00:32.272093+0000","last_undegraded":"2024-09-22T03:00:32.272093+0000","last_fullsized":"2024-09-22T03:00:32.272093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103022592,"num_objects":1572,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1572,"num_write_kb":100608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":48,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:26.195590+0000","last_change":"2024-09-22T03:00:26.196363+0000","last_active":"2024-09-22T03:00:26.195590+0000","last_peered":"2024-09-22T03:00:26.195590+0000","last_clean":"2024-09-22T02:59:44.153038+0000","last_became_active":"2024-09-22T03:00:26.191889+0000","last_became_peered":"2024-09-22T03:00:26.191889+0000","last_unstale":"2024-09-22T03:00:26.195590+0000","last_undegraded":"2024-09-22T03:00:26.195590+0000","last_fullsized":"2024-09-22T03:00:26.195590+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":173,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:26.165610+0000","last_change":"2024-09-22T03:00:26.166473+0000","last_active":"2024-09-22T03:00:26.165610+0000","last_peered":"2024-09-22T03:00:26.165610+0000","last_clean":"2024-09-22T02:59:44.153194+0000","last_became_active":"2024-09-22T03:00:26.159790+0000","last_became_peered":"2024-09-22T03:00:26.159790+0000","last_unstale":"2024-09-22T03:00:26.165610+0000","last_undegraded":"2024-09-22T03:00:26.165610+0000","last_fullsized":"2024-09-22T03:00:26.165610+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"24'1722","reported_seq":1764,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.295362+0000","last_change":"2024-09-22T03:00:26.354667+0000","last_active":"2024-09-22T03:00:32.295362+0000","last_peered":"2024-09-22T03:00:32.295362+0000","last_clean":"2024-09-22T03:00:32.295362+0000","last_became_active":"2024-09-22T03:00:26.184456+0000","last_became_peered":"2024-09-22T03:00:26.184456+0000","last_unstale":"2024-09-22T03:00:32.295362+0000","last_undegraded":"2024-09-22T03:00:32.295362+0000","last_fullsized":"2024-09-22T03:00:32.295362+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1722,"log_dups_size":0,"ondisk_log_size":1722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112918528,"num_objects":1723,"num_object_clones":0,"num_object_copies":5169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1723,"num_write_kb":110272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"24'1644","reported_seq":1679,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.250962+0000","last_change":"2024-09-22T03:00:26.187441+0000","last_active":"2024-09-22T03:00:32.250962+0000","last_peered":"2024-09-22T03:00:32.250962+0000","last_clean":"2024-09-22T03:00:32.250962+0000","last_became_active":"2024-09-22T03:00:26.184306+0000","last_became_peered":"2024-09-22T03:00:26.184306+0000","last_unstale":"2024-09-22T03:00:32.250962+0000","last_undegraded":"2024-09-22T03:00:32.250962+0000","last_fullsized":"2024-09-22T03:00:32.250962+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1644,"log_dups_size":0,"ondisk_log_size":1644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107806720,"num_objects":1645,"num_object_clones":0,"num_object_copies":4935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1645,"num_write_kb":105280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:26.194131+0000","last_change":"2024-09-22T03:00:26.194601+0000","last_active":"2024-09-22T03:00:26.194131+0000","last_peered":"2024-09-22T03:00:26.194131+0000","last_clean":"2024-09-22T02:58:45.380489+0000","last_became_active":"2024-09-22T03:00:26.191193+0000","last_became_peered":"2024-09-22T03:00:26.191193+0000","last_unstale":"2024-09-22T03:00:26.194131+0000","last_undegraded":"2024-09-22T03:00:26.194131+0000","last_fullsized":"2024-09-22T03:00:26.194131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"24'1655","reported_seq":1695,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.283799+0000","last_change":"2024-09-22T03:00:26.327795+0000","last_active":"2024-09-22T03:00:32.283799+0000","last_peered":"2024-09-22T03:00:32.283799+0000","last_clean":"2024-09-22T03:00:32.283799+0000","last_became_active":"2024-09-22T03:00:26.190969+0000","last_became_peered":"2024-09-22T03:00:26.190969+0000","last_unstale":"2024-09-22T03:00:32.283799+0000","last_undegraded":"2024-09-22T03:00:32.283799+0000","last_fullsized":"2024-09-22T03:00:32.283799+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1655,"log_dups_size":0,"ondisk_log_size":1655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108527616,"num_objects":1656,"num_object_clones":0,"num_object_copies":4968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1656,"num_write_kb":105984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:26.108917+0000","last_change":"2024-09-22T02:58:47.347987+0000","last_active":"2024-09-22T03:00:26.108917+0000","last_peered":"2024-09-22T03:00:26.108917+0000","last_clean":"2024-09-22T03:00:26.108917+0000","last_became_active":"2024-09-22T02:58:45.364066+0000","last_became_peered":"2024-09-22T02:58:45.364066+0000","last_unstale":"2024-09-22T03:00:26.108917+0000","last_undegraded":"2024-09-22T03:00:26.108917+0000","last_fullsized":"2024-09-22T03:00:26.108917+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"24'1563","reported_seq":1597,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:32.240764+0000","last_change":"2024-09-22T02:58:55.740822+0000","last_active":"2024-09-22T03:00:32.240764+0000","last_peered":"2024-09-22T03:00:32.240764+0000","last_clean":"2024-09-22T03:00:32.240764+0000","last_became_active":"2024-09-22T02:58:55.733222+0000","last_became_peered":"2024-09-22T02:58:55.733222+0000","last_unstale":"2024-09-22T03:00:32.240764+0000","last_undegraded":"2024-09-22T03:00:32.240764+0000","last_fullsized":"2024-09-22T03:00:32.240764+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1563,"log_dups_size":0,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1564,"num_write_kb":100096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:26.234988+0000","last_change":"2024-09-22T03:00:26.235433+0000","last_active":"2024-09-22T03:00:26.234988+0000","last_peered":"2024-09-22T03:00:26.234988+0000","last_clean":"2024-09-22T02:58:45.391068+0000","last_became_active":"2024-09-22T03:00:26.232118+0000","last_became_peered":"2024-09-22T03:00:26.232118+0000","last_unstale":"2024-09-22T03:00:26.234988+0000","last_undegraded":"2024-09-22T03:00:26.234988+0000","last_fullsized":"2024-09-22T03:00:26.234988+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"24'1498","reported_seq":1532,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:30.862796+0000","last_change":"2024-09-22T03:00:26.223794+0000","last_active":"2024-09-22T03:00:30.862796+0000","last_peered":"2024-09-22T03:00:30.862796+0000","last_clean":"2024-09-22T03:00:30.862796+0000","last_became_active":"2024-09-22T03:00:26.218041+0000","last_became_peered":"2024-09-22T03:00:26.218041+0000","last_unstale":"2024-09-22T03:00:30.862796+0000","last_undegraded":"2024-09-22T03:00:30.862796+0000","last_fullsized":"2024-09-22T03:00:30.862796+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98238464,"num_objects":1499,"num_object_clones":0,"num_object_copies":4497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1499,"num_write_kb":95936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-22T03:00:26.206128+0000","last_change":"2024-09-22T03:00:26.206817+0000","last_active":"2024-09-22T03:00:26.206128+0000","last_peered":"2024-09-22T03:00:26.206128+0000","last_clean":"2024-09-22T02:58:45.393707+0000","last_became_active":"2024-09-22T03:00:26.202803+0000","last_became_peered":"2024-09-22T03:00:26.202803+0000","last_unstale":"2024-09-22T03:00:26.206128+0000","last_undegraded":"2024-09-22T03:00:26.206128+0000","last_fullsized":"2024-09-22T03:00:26.206128+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1680801792,"num_objects":25647,"num_object_clones":0,"num_object_copies":76941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25647,"num_write_kb":1641408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":315211776,"data_stored":5043388416,"data_compressed":239483736,"data_compressed_allocated":315211776,"data_compressed_original":5043388416,"omap_allocated":0,"internal_metadata":0},"log_size":25631,"ondisk_log_size":25631,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":146992,"kb_used_data":89632,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94224848,"statfs":{"total":96636764160,"available":96486244352,"internally_reserved":0,"allocated":91783168,"data_stored":1465319478,"data_compressed":69577127,"data_compressed_allocated":91578368,"data_compressed_original":1465253888,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114380,"kb_used_data":69500,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94257460,"statfs":{"total":96636764160,"available":96519639040,"internally_reserved":0,"allocated":71168000,"data_stored":1135084118,"data_compressed":53897900,"data_compressed_allocated":70942720,"data_compressed_original":1135001600,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":127912,"kb_used_data":70548,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94243928,"statfs":{"total":96636764160,"available":96505782272,"internally_reserved":0,"allocated":72241152,"data_stored":1152320067,"data_compressed":54716377,"data_compressed_allocated":72019968,"data_compressed_original":1152237568,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":110016,"kb_used_data":75760,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":94261824,"statfs":{"total":96636764160,"available":96524107776,"internally_reserved":0,"allocated":77578240,"data_stored":1237647958,"data_compressed":58768129,"data_compressed_allocated":77352960,"data_compressed_original":1237565440,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":72753152,"data_stored":1164050432,"data_compressed":55274586,"data_compressed_allocated":72753152,"data_compressed_original":1164050432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78057472,"data_stored":1248919552,"data_compressed":59304554,"data_compressed_allocated":78057472,"data_compressed_original":1248919552,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71770112,"data_stored":1148321792,"data_compressed":54527685,"data_compressed_allocated":71770112,"data_compressed_original":1148321792,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":92631040,"data_stored":1482096640,"data_compressed":70376911,"data_compressed_allocated":92631040,"data_compressed_original":1482096640,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:00:34.760 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 98 16 26560 26544 16.927 19.25 0.0758269 0.0590494 2024-09-22T03:00:35.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 99 16 26882 26866 16.9591 20.125 0.0449636 0.0589491 2024-09-22T03:00:36.320 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:00:36.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:36.439+0000 7f91ab717640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:36.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:36.447+0000 7f91ab717640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:36.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-22T03:00:36.760748+0000 min lat: 0.0084511 max lat: 29.7608 avg lat: 0.0588357 2024-09-22T03:00:36.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:00:36.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 100 16 27206 27190 16.992 20.25 0.0505514 0.0588357 2024-09-22T03:00:37.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 101 16 27535 27519 17.0274 20.5625 0.0460761 0.0587169 2024-09-22T03:00:38.689 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:00:38.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 102 16 27854 27838 17.0559 19.9375 0.0636401 0.0586042 2024-09-22T03:00:38.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:38.839+0000 7f6415d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:38.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:38.846+0000 7f6415d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:39.582 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-09-22T03:00:39.582 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:00:39.582 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 2, 3, 1] 2024-09-22T03:00:39.583 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:00:39.583 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:00:39.761 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 103 16 28123 28107 17.0535 16.8125 0.013941 0.0585336 2024-09-22T03:00:40.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:00:40.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 104 16 28123 28107 16.8895 0 - 0.0585336 2024-09-22T03:00:41.167 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:00:41.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:41.279+0000 7f2abca6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:41.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:00:41.283+0000 7f2abca6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:00:41.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 105 16 28123 28107 16.7287 0 - 0.0585336 2024-09-22T03:00:42.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 106 16 28123 28107 16.5709 0 - 0.0585336 2024-09-22T03:00:43.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 107 16 28123 28107 16.416 0 - 0.0585336 2024-09-22T03:00:44.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 108 16 28123 28107 16.264 0 - 0.0585336 2024-09-22T03:00:45.684 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:00:45.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 109 16 28123 28107 16.1148 0 - 0.0585336 2024-09-22T03:00:45.995 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-22T03:00:46.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 110 16 28123 28107 15.9683 0 - 0.0585336 2024-09-22T03:00:47.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 111 16 28123 28107 15.8244 0 - 0.0585336 2024-09-22T03:00:48.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 112 16 28123 28107 15.6831 0 - 0.0585336 2024-09-22T03:00:49.762 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 113 16 28123 28107 15.5443 0 - 0.0585336 2024-09-22T03:00:50.684 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 3, 1] 2024-09-22T03:00:50.685 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:00:50.685 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-22T03:00:50.685 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:00:50.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:00:50.688 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:00:50.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:00:50.763 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 114 16 28123 28107 15.408 0 - 0.0585336 2024-09-22T03:00:50.798 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:00:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:00:50.802 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:00:51.763 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 115 16 28123 28107 15.274 0 - 0.0585336 2024-09-22T03:00:52.763 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 116 16 28123 28107 15.1423 0 - 0.0585336 2024-09-22T03:00:53.763 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 117 16 28123 28107 15.0129 0 - 0.0585336 2024-09-22T03:00:54.771 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 118 16 28123 28107 14.8856 0 - 0.0585336 2024-09-22T03:00:55.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 119 16 28123 28107 14.7605 0 - 0.0585336 2024-09-22T03:00:55.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:00:55.900 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:00:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:00:55.903 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:00:56.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-22T03:00:56.763337+0000 min lat: 0.0084511 max lat: 29.7608 avg lat: 0.0585336 2024-09-22T03:00:56.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:00:56.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 120 16 28123 28107 14.6375 0 - 0.0585336 2024-09-22T03:00:57.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 121 16 28123 28107 14.5165 0 - 0.0585336 2024-09-22T03:00:58.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 122 16 28123 28107 14.3976 0 - 0.0585336 2024-09-22T03:00:59.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 123 16 28123 28107 14.2805 0 - 0.0585336 2024-09-22T03:01:00.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 124 16 28123 28107 14.1653 0 - 0.0585336 2024-09-22T03:01:00.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:01:01.002 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-22T03:01:01.003 INFO:teuthology.orchestra.run.smithi190.stdout: "ops_in_flight": [] 2024-09-22T03:01:01.003 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-22T03:01:01.094 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:01:01.159 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:01:01.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:01:01.764 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 125 16 28123 28107 14.052 0 - 0.0585336 2024-09-22T03:01:02.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 126 16 28123 28107 13.9405 0 - 0.0585336 2024-09-22T03:01:03.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:01:03.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:03.299+0000 7fd9070d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:03.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:03.306+0000 7fd9070d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:03.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 127 16 28248 28232 13.8922 0.325521 0.0953729 0.0719591 2024-09-22T03:01:04.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 128 16 28549 28533 13.9306 18.8125 0.0548237 0.0717665 2024-09-22T03:01:05.504 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:01:05.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:05.641+0000 7faf956b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:05.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:05.648+0000 7faf956b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:05.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 129 16 28850 28834 13.9685 18.8125 0.0307906 0.0715686 2024-09-22T03:01:06.012 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 3, 1, 2] 2024-09-22T03:01:06.012 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:01:06.012 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:01:06.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:01:06.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:06.160+0000 7f9ee22ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:06.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:06.170+0000 7f9ee22ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:06.227 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:01:06.227 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:01:06.227 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:01:06.227 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:01:06.227 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:01:06.227 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:01:06.227 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:01:06.227 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:01:06.227 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:01:06.228 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:01:06.228 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:01:06.228 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:01:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:01:06.231 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:01:06.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:01:06.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:06.351+0000 7f2867223640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:06.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:06.366+0000 7f2867223640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:01:06.497 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:01:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:01:06.500 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:01:06.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:01:06.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:06.641+0000 7fb0290d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:06.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:06.652+0000 7fb0290d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:06.706 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:01:06.706 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:01:06.706 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:01:06.706 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:01:06.706 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:01:06.706 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:01:06.706 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:01:06.706 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:01:06.707 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:01:06.707 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:01:06.707 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:01:06.707 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:01:06.710 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:01:06.710 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:01:06.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:01:06.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 130 16 29140 29124 14.0004 18.125 0.0698222 0.0714057 2024-09-22T03:01:06.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:06.862+0000 7f079df97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:06.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:06.874+0000 7f079df97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:01:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:01:06.903 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:01:06.904 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:01:07.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 131 16 29445 29429 14.0391 19.0625 0.0479107 0.0712112 2024-09-22T03:01:07.954 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:01:08.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:08.094+0000 7fbb33e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:08.104+0000 7fbb33e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:08.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 132 16 29757 29741 14.0804 19.5 0.0388398 0.071003 2024-09-22T03:01:09.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 133 16 30061 30045 14.1174 19 0.0488431 0.0708093 2024-09-22T03:01:10.383 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:01:10.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:10.513+0000 7f3b74e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:10.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:10.521+0000 7f3b74e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:10.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 134 16 30370 30354 14.1562 19.3125 0.0342628 0.0706207 2024-09-22T03:01:11.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 135 16 30661 30645 14.186 18.1875 0.063088 0.0704721 2024-09-22T03:01:11.904 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 3, 1, 2] 2024-09-22T03:01:11.905 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:01:11.905 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 3, 1, 2] 2024-09-22T03:01:11.905 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:01:11.905 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:01:12.002 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:01:12.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 136 16 30705 30689 14.1019 2.75 0.0604362 0.0704431 2024-09-22T03:01:12.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:01:12.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:12.907+0000 7f0a9ff57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:12.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:12.912+0000 7f0a9ff57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:13.765 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 137 16 30705 30689 13.999 0 - 0.0704431 2024-09-22T03:01:14.766 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 138 16 30705 30689 13.8975 0 - 0.0704431 2024-09-22T03:01:15.766 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 139 16 30705 30689 13.7975 0 - 0.0704431 2024-09-22T03:01:16.766 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-22T03:01:16.765650+0000 min lat: 0.0084511 max lat: 29.7608 avg lat: 0.0704431 2024-09-22T03:01:16.766 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:01:16.766 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 140 16 30705 30689 13.699 0 - 0.0704431 2024-09-22T03:01:17.003 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2024-09-22T03:01:17.003 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-22T03:01:17.003 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:01:17.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:01:17.006 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:01:17.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:01:17.117 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:01:17.120 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:01:17.121 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:01:17.766 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 141 16 30705 30689 13.6018 0 - 0.0704431 2024-09-22T03:01:18.766 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 142 16 30705 30689 13.506 0 - 0.0704431 2024-09-22T03:01:19.766 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 143 16 30705 30689 13.4116 0 - 0.0704431 2024-09-22T03:01:20.767 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 144 16 30705 30689 13.3184 0 - 0.0704431 2024-09-22T03:01:21.767 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 145 16 30705 30689 13.2266 0 - 0.0704431 2024-09-22T03:01:22.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:01:22.218 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:01:22.221 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:01:22.221 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:01:22.767 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 146 16 30705 30689 13.136 0 - 0.0704431 2024-09-22T03:01:23.767 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 147 16 30705 30689 13.0466 0 - 0.0704431 2024-09-22T03:01:24.767 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 148 16 30705 30689 12.9585 0 - 0.0704431 2024-09-22T03:01:25.767 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 149 16 30705 30689 12.8715 0 - 0.0704431 2024-09-22T03:01:26.767 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 150 16 30705 30689 12.7857 0 - 0.0704431 2024-09-22T03:01:27.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:01:27.321 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:01:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:01:27.323 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:01:27.767 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 151 16 30705 30689 12.701 0 - 0.0704431 2024-09-22T03:01:28.768 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 152 16 30705 30689 12.6175 0 - 0.0704431 2024-09-22T03:01:29.768 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 153 16 30705 30689 12.535 0 - 0.0704431 2024-09-22T03:01:30.768 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 154 16 30705 30689 12.4536 0 - 0.0704431 2024-09-22T03:01:31.768 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 155 16 30705 30689 12.3732 0 - 0.0704431 2024-09-22T03:01:32.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:01:32.420 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:01:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:01:32.423 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:01:32.768 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 156 16 30705 30689 12.2939 0 - 0.0704431 2024-09-22T03:01:33.768 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 157 16 30705 30689 12.2156 0 - 0.0704431 2024-09-22T03:01:34.768 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 158 16 30705 30689 12.1383 0 - 0.0704431 2024-09-22T03:01:35.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 159 16 30705 30689 12.062 0 - 0.0704431 2024-09-22T03:01:36.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:2024-09-22T03:01:36.768367+0000 min lat: 0.0084511 max lat: 29.7608 avg lat: 0.0704431 2024-09-22T03:01:36.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:01:36.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 160 16 30705 30689 11.9866 0 - 0.0704431 2024-09-22T03:01:37.100 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:01:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:01:37.105 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:01:37.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:01:37.523 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-22T03:01:37.523 INFO:teuthology.orchestra.run.smithi190.stdout: "ops_in_flight": [] 2024-09-22T03:01:37.523 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-22T03:01:37.532 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-22T03:01:37.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:37.635+0000 7ff677797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:37.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:37.643+0000 7ff677797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:37.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 161 16 30705 30689 11.9121 0 - 0.0704431 2024-09-22T03:01:38.095 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.0 to 1 (10000) 2024-09-22T03:01:38.115 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-22T03:01:38.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:38.235+0000 7f16186ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:38.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:38.244+0000 7f16186ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:38.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout: 162 16 30705 30689 11.8386 0 - 0.0704431 2024-09-22T03:01:39.097 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.1 to 1 (10000) 2024-09-22T03:01:39.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:01:39.117 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-22T03:01:39.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:39.261+0000 7ff7c0957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:39.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:39.267+0000 7ff7c0957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:39.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:39.296+0000 7f8f94997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:39.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:39.308+0000 7f8f94997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:39.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total time run: 162.544 2024-09-22T03:01:39.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Total writes made: 30705 2024-09-22T03:01:39.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Write size: 65536 2024-09-22T03:01:39.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Object size: 65536 2024-09-22T03:01:39.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Bandwidth (MB/sec): 11.8064 2024-09-22T03:01:39.769 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Bandwidth: 11.9599 2024-09-22T03:01:39.770 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max bandwidth (MB/sec): 29.875 2024-09-22T03:01:39.770 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min bandwidth (MB/sec): 0 2024-09-22T03:01:39.770 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average IOPS: 188 2024-09-22T03:01:39.770 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev IOPS: 191.363 2024-09-22T03:01:39.770 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max IOPS: 478 2024-09-22T03:01:39.770 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min IOPS: 0 2024-09-22T03:01:39.770 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Average Latency(s): 0.0846673 2024-09-22T03:01:39.770 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Stddev Latency(s): 1.06666 2024-09-22T03:01:39.770 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Max latency(s): 29.7608 2024-09-22T03:01:39.770 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Min latency(s): 0.0084511 2024-09-22T03:01:39.785 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Cleaning up (deleting benchmark objects) 2024-09-22T03:01:40.100 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.2 to 1 (10000) 2024-09-22T03:01:40.117 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-22T03:01:40.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:40.254+0000 7f5f0eb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:40.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:40.268+0000 7f5f0eb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:41.101 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.3 to 1 (10000) 2024-09-22T03:01:41.119 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-22T03:01:41.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:01:41.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:41.240+0000 7f58ce997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:41.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:41.252+0000 7f58ce997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:41.364 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:01:41.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:41.508+0000 7fbf26797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:41.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:41.515+0000 7fbf26797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:41.532 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:01:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:01:41.548 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-09-22T03:01:41.102580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1988756043,"num_objects":30342,"num_object_clones":0,"num_object_copies":91026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":30669,"num_write_kb":1949609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30619,"ondisk_log_size":30619,"up":71,"acting":71,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":550972,"kb_used_data":363324,"kb_used_omap":0,"kb_used_meta":187584,"kb_avail":376936388,"statfs":{"total":386547056640,"available":385982861312,"internally_reserved":0,"allocated":372043776,"data_stored":5936638051,"data_compressed":281890766,"data_compressed_allocated":371032064,"data_compressed_original":5936267264,"omap_allocated":0,"internal_metadata":192086016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-655336,"num_objects":-9,"num_object_clones":0,"num_object_copies":-27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183,"num_write_kb":5505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008711"},"pg_stats":[{"pgid":"3.f","version":"27'1941","reported_seq":1992,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:11.800110+0000","last_change":"2024-09-22T03:01:37.120882+0000","last_active":"2024-09-22T03:01:37.120882+0000","last_peered":"2024-09-22T03:01:11.800110+0000","last_clean":"2024-09-22T03:01:11.800110+0000","last_became_active":"2024-09-22T03:00:26.186244+0000","last_became_peered":"2024-09-22T03:00:26.186244+0000","last_unstale":"2024-09-22T03:01:11.800110+0000","last_undegraded":"2024-09-22T03:01:11.800110+0000","last_fullsized":"2024-09-22T03:01:11.800110+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1941,"log_dups_size":0,"ondisk_log_size":1941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127205376,"num_objects":1941,"num_object_clones":0,"num_object_copies":5823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1941,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'1912","reported_seq":1961,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:11.814934+0000","last_change":"2024-09-22T03:01:03.244236+0000","last_active":"2024-09-22T03:01:11.814934+0000","last_peered":"2024-09-22T03:01:11.814934+0000","last_clean":"2024-09-22T03:01:11.814934+0000","last_became_active":"2024-09-22T03:01:03.238574+0000","last_became_peered":"2024-09-22T03:01:03.238574+0000","last_unstale":"2024-09-22T03:01:11.814934+0000","last_undegraded":"2024-09-22T03:01:11.814934+0000","last_fullsized":"2024-09-22T03:01:11.814934+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1912,"log_dups_size":0,"ondisk_log_size":1912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125370368,"num_objects":1913,"num_object_clones":0,"num_object_copies":5739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1913,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'1947","reported_seq":2009,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.808541+0000","last_change":"2024-09-22T03:01:39.304739+0000","last_active":"2024-09-22T03:01:40.808541+0000","last_peered":"2024-09-22T03:01:40.808541+0000","last_clean":"2024-09-22T03:01:40.808541+0000","last_became_active":"2024-09-22T03:01:39.272458+0000","last_became_peered":"2024-09-22T03:01:39.272458+0000","last_unstale":"2024-09-22T03:01:40.808541+0000","last_undegraded":"2024-09-22T03:01:40.808541+0000","last_fullsized":"2024-09-22T03:01:40.808541+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1947,"log_dups_size":0,"ondisk_log_size":1947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124977176,"num_objects":1908,"num_object_clones":0,"num_object_copies":5724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1948,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'1847","reported_seq":1896,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:11.893913+0000","last_change":"2024-09-22T03:01:37.121638+0000","last_active":"2024-09-22T03:01:37.121638+0000","last_peered":"2024-09-22T03:01:11.893913+0000","last_clean":"2024-09-22T03:01:11.893913+0000","last_became_active":"2024-09-22T03:01:03.195863+0000","last_became_peered":"2024-09-22T03:01:03.195863+0000","last_unstale":"2024-09-22T03:01:11.893913+0000","last_undegraded":"2024-09-22T03:01:11.893913+0000","last_fullsized":"2024-09-22T03:01:11.893913+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1847,"log_dups_size":0,"ondisk_log_size":1847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121044992,"num_objects":1847,"num_object_clones":0,"num_object_copies":5541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1847,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'1904","reported_seq":1955,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:11.891523+0000","last_change":"2024-09-22T03:01:37.121582+0000","last_active":"2024-09-22T03:01:37.121582+0000","last_peered":"2024-09-22T03:01:11.891523+0000","last_clean":"2024-09-22T03:01:11.891523+0000","last_became_active":"2024-09-22T03:01:03.216120+0000","last_became_peered":"2024-09-22T03:01:03.216120+0000","last_unstale":"2024-09-22T03:01:11.891523+0000","last_undegraded":"2024-09-22T03:01:11.891523+0000","last_fullsized":"2024-09-22T03:01:11.891523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124780544,"num_objects":1904,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":121856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"27'1895","reported_seq":1949,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.118236+0000","last_change":"2024-09-22T03:01:03.193486+0000","last_active":"2024-09-22T03:01:37.118236+0000","last_peered":"2024-09-22T03:01:37.118236+0000","last_clean":"2024-09-22T03:01:37.118236+0000","last_became_active":"2024-09-22T03:01:03.189517+0000","last_became_peered":"2024-09-22T03:01:03.189517+0000","last_unstale":"2024-09-22T03:01:37.118236+0000","last_undegraded":"2024-09-22T03:01:37.118236+0000","last_fullsized":"2024-09-22T03:01:37.118236+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1895,"log_dups_size":0,"ondisk_log_size":1895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124190720,"num_objects":1895,"num_object_clones":0,"num_object_copies":5685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1895,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'1865","reported_seq":1917,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.123451+0000","last_change":"2024-09-22T03:01:03.363540+0000","last_active":"2024-09-22T03:01:37.123451+0000","last_peered":"2024-09-22T03:01:37.123451+0000","last_clean":"2024-09-22T03:01:37.123451+0000","last_became_active":"2024-09-22T03:01:03.209817+0000","last_became_peered":"2024-09-22T03:01:03.209817+0000","last_unstale":"2024-09-22T03:01:37.123451+0000","last_undegraded":"2024-09-22T03:01:37.123451+0000","last_fullsized":"2024-09-22T03:01:37.123451+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1865,"log_dups_size":0,"ondisk_log_size":1865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122224640,"num_objects":1865,"num_object_clones":0,"num_object_copies":5595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'1969","reported_seq":2031,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.819231+0000","last_change":"2024-09-22T03:01:39.209087+0000","last_active":"2024-09-22T03:01:40.819231+0000","last_peered":"2024-09-22T03:01:40.819231+0000","last_clean":"2024-09-22T03:01:40.819231+0000","last_became_active":"2024-09-22T03:01:39.196396+0000","last_became_peered":"2024-09-22T03:01:39.196396+0000","last_unstale":"2024-09-22T03:01:40.819231+0000","last_undegraded":"2024-09-22T03:01:40.819231+0000","last_fullsized":"2024-09-22T03:01:40.819231+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1969,"log_dups_size":0,"ondisk_log_size":1969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125304832,"num_objects":1912,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1970,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:03.143590+0000","last_change":"2024-09-22T03:01:01.102795+0000","last_active":"2024-09-22T03:01:01.102795+0000","last_peered":"2024-09-22T03:00:26.124572+0000","last_clean":"2024-09-22T03:00:26.124572+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T03:01:03.143590+0000","last_undegraded":"2024-09-22T03:01:03.143590+0000","last_fullsized":"2024-09-22T03:01:03.143590+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"31'1957","reported_seq":2013,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.832771+0000","last_change":"2024-09-22T03:01:39.198947+0000","last_active":"2024-09-22T03:01:40.832771+0000","last_peered":"2024-09-22T03:01:40.832771+0000","last_clean":"2024-09-22T03:01:40.832771+0000","last_became_active":"2024-09-22T03:01:39.186612+0000","last_became_peered":"2024-09-22T03:01:39.186612+0000","last_unstale":"2024-09-22T03:01:40.832771+0000","last_undegraded":"2024-09-22T03:01:40.832771+0000","last_fullsized":"2024-09-22T03:01:40.832771+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1957,"log_dups_size":0,"ondisk_log_size":1957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125042688,"num_objects":1908,"num_object_clones":0,"num_object_copies":5724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1958,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.109344+0000","last_change":"2024-09-22T03:01:37.119403+0000","last_active":"2024-09-22T03:01:37.109344+0000","last_peered":"2024-09-22T03:01:37.109344+0000","last_clean":"2024-09-22T03:01:37.109344+0000","last_became_active":"2024-09-22T03:01:03.184670+0000","last_became_peered":"2024-09-22T03:01:03.184670+0000","last_unstale":"2024-09-22T03:01:37.109344+0000","last_undegraded":"2024-09-22T03:01:37.109344+0000","last_fullsized":"2024-09-22T03:01:37.109344+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'1958","reported_seq":2001,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:11.847340+0000","last_change":"2024-09-22T03:01:03.261552+0000","last_active":"2024-09-22T03:01:11.847340+0000","last_peered":"2024-09-22T03:01:11.847340+0000","last_clean":"2024-09-22T03:01:11.847340+0000","last_became_active":"2024-09-22T03:01:03.250667+0000","last_became_peered":"2024-09-22T03:01:03.250667+0000","last_unstale":"2024-09-22T03:01:11.847340+0000","last_undegraded":"2024-09-22T03:01:11.847340+0000","last_fullsized":"2024-09-22T03:01:11.847340+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1958,"log_dups_size":0,"ondisk_log_size":1958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128385024,"num_objects":1959,"num_object_clones":0,"num_object_copies":5877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:03.275677+0000","last_change":"2024-09-22T03:01:03.276187+0000","last_active":"2024-09-22T03:01:03.275677+0000","last_peered":"2024-09-22T03:01:03.275677+0000","last_clean":"2024-09-22T03:00:26.171111+0000","last_became_active":"2024-09-22T03:01:03.254209+0000","last_became_peered":"2024-09-22T03:01:03.254209+0000","last_unstale":"2024-09-22T03:01:03.275677+0000","last_undegraded":"2024-09-22T03:01:03.275677+0000","last_fullsized":"2024-09-22T03:01:03.275677+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"27'1801","reported_seq":1858,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:06.858615+0000","last_change":"2024-09-22T03:01:03.365285+0000","last_active":"2024-09-22T03:01:06.858615+0000","last_peered":"2024-09-22T03:01:06.858615+0000","last_clean":"2024-09-22T03:01:06.858615+0000","last_became_active":"2024-09-22T03:01:03.180158+0000","last_became_peered":"2024-09-22T03:01:03.180158+0000","last_unstale":"2024-09-22T03:01:37.093892+0000","last_undegraded":"2024-09-22T03:01:06.858615+0000","last_fullsized":"2024-09-22T03:01:06.858615+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1801,"log_dups_size":0,"ondisk_log_size":1801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1802,"num_write_kb":115328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":58,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:03.148641+0000","last_change":"2024-09-22T03:01:01.124723+0000","last_active":"2024-09-22T03:01:03.148641+0000","last_peered":"2024-09-22T03:01:03.148641+0000","last_clean":"2024-09-22T03:01:03.148641+0000","last_became_active":"2024-09-22T03:00:26.191889+0000","last_became_peered":"2024-09-22T03:00:26.191889+0000","last_unstale":"2024-09-22T03:01:37.093893+0000","last_undegraded":"2024-09-22T03:01:03.148641+0000","last_fullsized":"2024-09-22T03:01:03.148641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":186,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.112908+0000","last_change":"2024-09-22T03:01:03.199665+0000","last_active":"2024-09-22T03:01:37.112908+0000","last_peered":"2024-09-22T03:01:37.112908+0000","last_clean":"2024-09-22T03:01:37.112908+0000","last_became_active":"2024-09-22T03:01:03.194697+0000","last_became_peered":"2024-09-22T03:01:03.194697+0000","last_unstale":"2024-09-22T03:01:37.112908+0000","last_undegraded":"2024-09-22T03:01:37.112908+0000","last_fullsized":"2024-09-22T03:01:37.112908+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'1944","reported_seq":1995,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:07.273651+0000","last_change":"2024-09-22T03:00:26.354667+0000","last_active":"2024-09-22T03:01:07.273651+0000","last_peered":"2024-09-22T03:01:07.273651+0000","last_clean":"2024-09-22T03:01:07.273651+0000","last_became_active":"2024-09-22T03:00:26.184456+0000","last_became_peered":"2024-09-22T03:00:26.184456+0000","last_unstale":"2024-09-22T03:01:37.093894+0000","last_undegraded":"2024-09-22T03:01:07.273651+0000","last_fullsized":"2024-09-22T03:01:07.273651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1944,"log_dups_size":0,"ondisk_log_size":1944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127467520,"num_objects":1945,"num_object_clones":0,"num_object_copies":5835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1945,"num_write_kb":124480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"27'1961","reported_seq":2005,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:11.787922+0000","last_change":"2024-09-22T03:01:03.274707+0000","last_active":"2024-09-22T03:01:11.787922+0000","last_peered":"2024-09-22T03:01:11.787922+0000","last_clean":"2024-09-22T03:01:11.787922+0000","last_became_active":"2024-09-22T03:01:03.257726+0000","last_became_peered":"2024-09-22T03:01:03.257726+0000","last_unstale":"2024-09-22T03:01:11.787922+0000","last_undegraded":"2024-09-22T03:01:11.787922+0000","last_fullsized":"2024-09-22T03:01:11.787922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1961,"log_dups_size":0,"ondisk_log_size":1961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128581632,"num_objects":1962,"num_object_clones":0,"num_object_copies":5886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1962,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.136051+0000","last_change":"2024-09-22T03:01:37.133459+0000","last_active":"2024-09-22T03:01:40.136051+0000","last_peered":"2024-09-22T03:01:40.136051+0000","last_clean":"2024-09-22T03:01:40.136051+0000","last_became_active":"2024-09-22T03:01:03.184548+0000","last_became_peered":"2024-09-22T03:01:03.184548+0000","last_unstale":"2024-09-22T03:01:40.136051+0000","last_undegraded":"2024-09-22T03:01:40.136051+0000","last_fullsized":"2024-09-22T03:01:40.136051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"27'1883","reported_seq":1932,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:07.306661+0000","last_change":"2024-09-22T03:00:26.327795+0000","last_active":"2024-09-22T03:01:07.306661+0000","last_peered":"2024-09-22T03:01:07.306661+0000","last_clean":"2024-09-22T03:01:07.306661+0000","last_became_active":"2024-09-22T03:00:26.190969+0000","last_became_peered":"2024-09-22T03:00:26.190969+0000","last_unstale":"2024-09-22T03:01:37.093896+0000","last_undegraded":"2024-09-22T03:01:07.306661+0000","last_fullsized":"2024-09-22T03:01:07.306661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1883,"log_dups_size":0,"ondisk_log_size":1883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123469824,"num_objects":1884,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1884,"num_write_kb":120576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:03.231131+0000","last_change":"2024-09-22T03:01:03.231788+0000","last_active":"2024-09-22T03:01:03.231131+0000","last_peered":"2024-09-22T03:01:03.231131+0000","last_clean":"2024-09-22T02:58:45.368663+0000","last_became_active":"2024-09-22T03:01:03.226859+0000","last_became_peered":"2024-09-22T03:01:03.226859+0000","last_unstale":"2024-09-22T03:01:03.231131+0000","last_undegraded":"2024-09-22T03:01:03.231131+0000","last_fullsized":"2024-09-22T03:01:03.231131+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'1892","reported_seq":1935,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:11.778507+0000","last_change":"2024-09-22T03:01:03.217552+0000","last_active":"2024-09-22T03:01:11.778507+0000","last_peered":"2024-09-22T03:01:11.778507+0000","last_clean":"2024-09-22T03:01:11.778507+0000","last_became_active":"2024-09-22T03:01:03.212561+0000","last_became_peered":"2024-09-22T03:01:03.212561+0000","last_unstale":"2024-09-22T03:01:11.778507+0000","last_undegraded":"2024-09-22T03:01:11.778507+0000","last_fullsized":"2024-09-22T03:01:11.778507+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1892,"log_dups_size":0,"ondisk_log_size":1892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124059648,"num_objects":1893,"num_object_clones":0,"num_object_copies":5679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1893,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.136294+0000","last_change":"2024-09-22T03:01:40.155884+0000","last_active":"2024-09-22T03:01:40.136294+0000","last_peered":"2024-09-22T03:01:40.136294+0000","last_clean":"2024-09-22T03:01:40.136294+0000","last_became_active":"2024-09-22T03:01:39.201350+0000","last_became_peered":"2024-09-22T03:01:39.201350+0000","last_unstale":"2024-09-22T03:01:40.136294+0000","last_undegraded":"2024-09-22T03:01:40.136294+0000","last_fullsized":"2024-09-22T03:01:40.136294+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"31'1843","reported_seq":1900,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.660073+0000","last_change":"2024-09-22T03:01:39.201120+0000","last_active":"2024-09-22T03:01:40.660073+0000","last_peered":"2024-09-22T03:01:40.660073+0000","last_clean":"2024-09-22T03:01:40.660073+0000","last_became_active":"2024-09-22T03:01:39.191673+0000","last_became_peered":"2024-09-22T03:01:39.191673+0000","last_unstale":"2024-09-22T03:01:40.660073+0000","last_undegraded":"2024-09-22T03:01:40.660073+0000","last_fullsized":"2024-09-22T03:01:40.660073+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1843,"log_dups_size":0,"ondisk_log_size":1843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117964800,"num_objects":1800,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1844,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.135447+0000","last_change":"2024-09-22T03:01:40.155556+0000","last_active":"2024-09-22T03:01:40.135447+0000","last_peered":"2024-09-22T03:01:40.135447+0000","last_clean":"2024-09-22T03:01:40.135447+0000","last_became_active":"2024-09-22T03:01:39.204853+0000","last_became_peered":"2024-09-22T03:01:39.204853+0000","last_unstale":"2024-09-22T03:01:40.135447+0000","last_undegraded":"2024-09-22T03:01:40.135447+0000","last_fullsized":"2024-09-22T03:01:40.135447+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1988165656,"num_objects":30338,"num_object_clones":0,"num_object_copies":91014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":30530,"num_write_kb":1947713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":371122176,"data_stored":5937889304,"data_compressed":281959780,"data_compressed_allocated":371118080,"data_compressed_original":5937889280,"omap_allocated":0,"internal_metadata":0},"log_size":30519,"ondisk_log_size":30519,"up":45,"acting":45,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":23,"acting":23,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673104,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":155344,"kb_used_data":102144,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94216496,"statfs":{"total":96636764160,"available":96477691904,"internally_reserved":0,"allocated":104595456,"data_stored":1669933723,"data_compressed":79292791,"data_compressed_allocated":104366080,"data_compressed_original":1669857280,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":26,"seq":111669149730,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126652,"kb_used_data":83820,"kb_used_omap":0,"kb_used_meta":42816,"kb_avail":94245188,"statfs":{"total":96636764160,"available":96507072512,"internally_reserved":0,"allocated":85831680,"data_stored":1369320635,"data_compressed":65020188,"data_compressed_allocated":85581824,"data_compressed_original":1369227264,"omap_allocated":0,"internal_metadata":43843584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":142356,"kb_used_data":84996,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94229484,"statfs":{"total":96636764160,"available":96490991616,"internally_reserved":0,"allocated":87035904,"data_stored":1388653736,"data_compressed":65938249,"data_compressed_allocated":86790144,"data_compressed_original":1388560384,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126620,"kb_used_data":92364,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":94245220,"statfs":{"total":96636764160,"available":96507105280,"internally_reserved":0,"allocated":94580736,"data_stored":1508729957,"data_compressed":71639538,"data_compressed_allocated":94294016,"data_compressed_original":1508622336,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86749184,"data_stored":1387986944,"data_compressed":65908290,"data_compressed_allocated":86749184,"data_compressed_original":1387986944,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":93523968,"data_stored":1496317976,"data_compressed":71052710,"data_compressed_allocated":93519872,"data_compressed_original":1496317952,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85540864,"data_stored":1368653824,"data_compressed":64990229,"data_compressed_allocated":85540864,"data_compressed_original":1368653824,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":105308160,"data_stored":1684930560,"data_compressed":80008551,"data_compressed_allocated":105308160,"data_compressed_original":1684930560,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:01:41.550 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:01:41.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:41.689+0000 7fd1f9597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:41.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:41.699+0000 7fd1f9597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:42.051 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:01:42.051 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:01:42.066 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-09-22T03:01:41.102580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1988756043,"num_objects":30342,"num_object_clones":0,"num_object_copies":91026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":30669,"num_write_kb":1949609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30619,"ondisk_log_size":30619,"up":71,"acting":71,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":550972,"kb_used_data":363324,"kb_used_omap":0,"kb_used_meta":187584,"kb_avail":376936388,"statfs":{"total":386547056640,"available":385982861312,"internally_reserved":0,"allocated":372043776,"data_stored":5936638051,"data_compressed":281890766,"data_compressed_allocated":371032064,"data_compressed_original":5936267264,"omap_allocated":0,"internal_metadata":192086016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-655336,"num_objects":-9,"num_object_clones":0,"num_object_copies":-27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183,"num_write_kb":5505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008711"},"pg_stats":[{"pgid":"3.f","version":"27'1941","reported_seq":1992,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:11.800110+0000","last_change":"2024-09-22T03:01:37.120882+0000","last_active":"2024-09-22T03:01:37.120882+0000","last_peered":"2024-09-22T03:01:11.800110+0000","last_clean":"2024-09-22T03:01:11.800110+0000","last_became_active":"2024-09-22T03:00:26.186244+0000","last_became_peered":"2024-09-22T03:00:26.186244+0000","last_unstale":"2024-09-22T03:01:11.800110+0000","last_undegraded":"2024-09-22T03:01:11.800110+0000","last_fullsized":"2024-09-22T03:01:11.800110+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1941,"log_dups_size":0,"ondisk_log_size":1941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127205376,"num_objects":1941,"num_object_clones":0,"num_object_copies":5823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1941,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'1912","reported_seq":1961,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:11.814934+0000","last_change":"2024-09-22T03:01:03.244236+0000","last_active":"2024-09-22T03:01:11.814934+0000","last_peered":"2024-09-22T03:01:11.814934+0000","last_clean":"2024-09-22T03:01:11.814934+0000","last_became_active":"2024-09-22T03:01:03.238574+0000","last_became_peered":"2024-09-22T03:01:03.238574+0000","last_unstale":"2024-09-22T03:01:11.814934+0000","last_undegraded":"2024-09-22T03:01:11.814934+0000","last_fullsized":"2024-09-22T03:01:11.814934+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1912,"log_dups_size":0,"ondisk_log_size":1912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125370368,"num_objects":1913,"num_object_clones":0,"num_object_copies":5739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1913,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'1947","reported_seq":2009,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.808541+0000","last_change":"2024-09-22T03:01:39.304739+0000","last_active":"2024-09-22T03:01:40.808541+0000","last_peered":"2024-09-22T03:01:40.808541+0000","last_clean":"2024-09-22T03:01:40.808541+0000","last_became_active":"2024-09-22T03:01:39.272458+0000","last_became_peered":"2024-09-22T03:01:39.272458+0000","last_unstale":"2024-09-22T03:01:40.808541+0000","last_undegraded":"2024-09-22T03:01:40.808541+0000","last_fullsized":"2024-09-22T03:01:40.808541+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1947,"log_dups_size":0,"ondisk_log_size":1947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124977176,"num_objects":1908,"num_object_clones":0,"num_object_copies":5724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1948,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'1847","reported_seq":1896,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:11.893913+0000","last_change":"2024-09-22T03:01:37.121638+0000","last_active":"2024-09-22T03:01:37.121638+0000","last_peered":"2024-09-22T03:01:11.893913+0000","last_clean":"2024-09-22T03:01:11.893913+0000","last_became_active":"2024-09-22T03:01:03.195863+0000","last_became_peered":"2024-09-22T03:01:03.195863+0000","last_unstale":"2024-09-22T03:01:11.893913+0000","last_undegraded":"2024-09-22T03:01:11.893913+0000","last_fullsized":"2024-09-22T03:01:11.893913+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1847,"log_dups_size":0,"ondisk_log_size":1847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121044992,"num_objects":1847,"num_object_clones":0,"num_object_copies":5541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1847,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'1904","reported_seq":1955,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:11.891523+0000","last_change":"2024-09-22T03:01:37.121582+0000","last_active":"2024-09-22T03:01:37.121582+0000","last_peered":"2024-09-22T03:01:11.891523+0000","last_clean":"2024-09-22T03:01:11.891523+0000","last_became_active":"2024-09-22T03:01:03.216120+0000","last_became_peered":"2024-09-22T03:01:03.216120+0000","last_unstale":"2024-09-22T03:01:11.891523+0000","last_undegraded":"2024-09-22T03:01:11.891523+0000","last_fullsized":"2024-09-22T03:01:11.891523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124780544,"num_objects":1904,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":121856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"27'1895","reported_seq":1949,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.118236+0000","last_change":"2024-09-22T03:01:03.193486+0000","last_active":"2024-09-22T03:01:37.118236+0000","last_peered":"2024-09-22T03:01:37.118236+0000","last_clean":"2024-09-22T03:01:37.118236+0000","last_became_active":"2024-09-22T03:01:03.189517+0000","last_became_peered":"2024-09-22T03:01:03.189517+0000","last_unstale":"2024-09-22T03:01:37.118236+0000","last_undegraded":"2024-09-22T03:01:37.118236+0000","last_fullsized":"2024-09-22T03:01:37.118236+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1895,"log_dups_size":0,"ondisk_log_size":1895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124190720,"num_objects":1895,"num_object_clones":0,"num_object_copies":5685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1895,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'1865","reported_seq":1917,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.123451+0000","last_change":"2024-09-22T03:01:03.363540+0000","last_active":"2024-09-22T03:01:37.123451+0000","last_peered":"2024-09-22T03:01:37.123451+0000","last_clean":"2024-09-22T03:01:37.123451+0000","last_became_active":"2024-09-22T03:01:03.209817+0000","last_became_peered":"2024-09-22T03:01:03.209817+0000","last_unstale":"2024-09-22T03:01:37.123451+0000","last_undegraded":"2024-09-22T03:01:37.123451+0000","last_fullsized":"2024-09-22T03:01:37.123451+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1865,"log_dups_size":0,"ondisk_log_size":1865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122224640,"num_objects":1865,"num_object_clones":0,"num_object_copies":5595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'1969","reported_seq":2031,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.819231+0000","last_change":"2024-09-22T03:01:39.209087+0000","last_active":"2024-09-22T03:01:40.819231+0000","last_peered":"2024-09-22T03:01:40.819231+0000","last_clean":"2024-09-22T03:01:40.819231+0000","last_became_active":"2024-09-22T03:01:39.196396+0000","last_became_peered":"2024-09-22T03:01:39.196396+0000","last_unstale":"2024-09-22T03:01:40.819231+0000","last_undegraded":"2024-09-22T03:01:40.819231+0000","last_fullsized":"2024-09-22T03:01:40.819231+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1969,"log_dups_size":0,"ondisk_log_size":1969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125304832,"num_objects":1912,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1970,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:03.143590+0000","last_change":"2024-09-22T03:01:01.102795+0000","last_active":"2024-09-22T03:01:01.102795+0000","last_peered":"2024-09-22T03:00:26.124572+0000","last_clean":"2024-09-22T03:00:26.124572+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T03:01:03.143590+0000","last_undegraded":"2024-09-22T03:01:03.143590+0000","last_fullsized":"2024-09-22T03:01:03.143590+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"31'1957","reported_seq":2013,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.832771+0000","last_change":"2024-09-22T03:01:39.198947+0000","last_active":"2024-09-22T03:01:40.832771+0000","last_peered":"2024-09-22T03:01:40.832771+0000","last_clean":"2024-09-22T03:01:40.832771+0000","last_became_active":"2024-09-22T03:01:39.186612+0000","last_became_peered":"2024-09-22T03:01:39.186612+0000","last_unstale":"2024-09-22T03:01:40.832771+0000","last_undegraded":"2024-09-22T03:01:40.832771+0000","last_fullsized":"2024-09-22T03:01:40.832771+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1957,"log_dups_size":0,"ondisk_log_size":1957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125042688,"num_objects":1908,"num_object_clones":0,"num_object_copies":5724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1958,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.109344+0000","last_change":"2024-09-22T03:01:37.119403+0000","last_active":"2024-09-22T03:01:37.109344+0000","last_peered":"2024-09-22T03:01:37.109344+0000","last_clean":"2024-09-22T03:01:37.109344+0000","last_became_active":"2024-09-22T03:01:03.184670+0000","last_became_peered":"2024-09-22T03:01:03.184670+0000","last_unstale":"2024-09-22T03:01:37.109344+0000","last_undegraded":"2024-09-22T03:01:37.109344+0000","last_fullsized":"2024-09-22T03:01:37.109344+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'1958","reported_seq":2001,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:11.847340+0000","last_change":"2024-09-22T03:01:03.261552+0000","last_active":"2024-09-22T03:01:11.847340+0000","last_peered":"2024-09-22T03:01:11.847340+0000","last_clean":"2024-09-22T03:01:11.847340+0000","last_became_active":"2024-09-22T03:01:03.250667+0000","last_became_peered":"2024-09-22T03:01:03.250667+0000","last_unstale":"2024-09-22T03:01:11.847340+0000","last_undegraded":"2024-09-22T03:01:11.847340+0000","last_fullsized":"2024-09-22T03:01:11.847340+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1958,"log_dups_size":0,"ondisk_log_size":1958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128385024,"num_objects":1959,"num_object_clones":0,"num_object_copies":5877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":58,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:03.275677+0000","last_change":"2024-09-22T03:01:03.276187+0000","last_active":"2024-09-22T03:01:03.275677+0000","last_peered":"2024-09-22T03:01:03.275677+0000","last_clean":"2024-09-22T03:00:26.171111+0000","last_became_active":"2024-09-22T03:01:03.254209+0000","last_became_peered":"2024-09-22T03:01:03.254209+0000","last_unstale":"2024-09-22T03:01:03.275677+0000","last_undegraded":"2024-09-22T03:01:03.275677+0000","last_fullsized":"2024-09-22T03:01:03.275677+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"27'1801","reported_seq":1858,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:06.858615+0000","last_change":"2024-09-22T03:01:03.365285+0000","last_active":"2024-09-22T03:01:06.858615+0000","last_peered":"2024-09-22T03:01:06.858615+0000","last_clean":"2024-09-22T03:01:06.858615+0000","last_became_active":"2024-09-22T03:01:03.180158+0000","last_became_peered":"2024-09-22T03:01:03.180158+0000","last_unstale":"2024-09-22T03:01:37.093892+0000","last_undegraded":"2024-09-22T03:01:06.858615+0000","last_fullsized":"2024-09-22T03:01:06.858615+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1801,"log_dups_size":0,"ondisk_log_size":1801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1802,"num_write_kb":115328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":58,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:03.148641+0000","last_change":"2024-09-22T03:01:01.124723+0000","last_active":"2024-09-22T03:01:03.148641+0000","last_peered":"2024-09-22T03:01:03.148641+0000","last_clean":"2024-09-22T03:01:03.148641+0000","last_became_active":"2024-09-22T03:00:26.191889+0000","last_became_peered":"2024-09-22T03:00:26.191889+0000","last_unstale":"2024-09-22T03:01:37.093893+0000","last_undegraded":"2024-09-22T03:01:03.148641+0000","last_fullsized":"2024-09-22T03:01:03.148641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":186,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.112908+0000","last_change":"2024-09-22T03:01:03.199665+0000","last_active":"2024-09-22T03:01:37.112908+0000","last_peered":"2024-09-22T03:01:37.112908+0000","last_clean":"2024-09-22T03:01:37.112908+0000","last_became_active":"2024-09-22T03:01:03.194697+0000","last_became_peered":"2024-09-22T03:01:03.194697+0000","last_unstale":"2024-09-22T03:01:37.112908+0000","last_undegraded":"2024-09-22T03:01:37.112908+0000","last_fullsized":"2024-09-22T03:01:37.112908+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'1944","reported_seq":1995,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:07.273651+0000","last_change":"2024-09-22T03:00:26.354667+0000","last_active":"2024-09-22T03:01:07.273651+0000","last_peered":"2024-09-22T03:01:07.273651+0000","last_clean":"2024-09-22T03:01:07.273651+0000","last_became_active":"2024-09-22T03:00:26.184456+0000","last_became_peered":"2024-09-22T03:00:26.184456+0000","last_unstale":"2024-09-22T03:01:37.093894+0000","last_undegraded":"2024-09-22T03:01:07.273651+0000","last_fullsized":"2024-09-22T03:01:07.273651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1944,"log_dups_size":0,"ondisk_log_size":1944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127467520,"num_objects":1945,"num_object_clones":0,"num_object_copies":5835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1945,"num_write_kb":124480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"27'1961","reported_seq":2005,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:11.787922+0000","last_change":"2024-09-22T03:01:03.274707+0000","last_active":"2024-09-22T03:01:11.787922+0000","last_peered":"2024-09-22T03:01:11.787922+0000","last_clean":"2024-09-22T03:01:11.787922+0000","last_became_active":"2024-09-22T03:01:03.257726+0000","last_became_peered":"2024-09-22T03:01:03.257726+0000","last_unstale":"2024-09-22T03:01:11.787922+0000","last_undegraded":"2024-09-22T03:01:11.787922+0000","last_fullsized":"2024-09-22T03:01:11.787922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1961,"log_dups_size":0,"ondisk_log_size":1961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128581632,"num_objects":1962,"num_object_clones":0,"num_object_copies":5886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1962,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.136051+0000","last_change":"2024-09-22T03:01:37.133459+0000","last_active":"2024-09-22T03:01:40.136051+0000","last_peered":"2024-09-22T03:01:40.136051+0000","last_clean":"2024-09-22T03:01:40.136051+0000","last_became_active":"2024-09-22T03:01:03.184548+0000","last_became_peered":"2024-09-22T03:01:03.184548+0000","last_unstale":"2024-09-22T03:01:40.136051+0000","last_undegraded":"2024-09-22T03:01:40.136051+0000","last_fullsized":"2024-09-22T03:01:40.136051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"27'1883","reported_seq":1932,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:07.306661+0000","last_change":"2024-09-22T03:00:26.327795+0000","last_active":"2024-09-22T03:01:07.306661+0000","last_peered":"2024-09-22T03:01:07.306661+0000","last_clean":"2024-09-22T03:01:07.306661+0000","last_became_active":"2024-09-22T03:00:26.190969+0000","last_became_peered":"2024-09-22T03:00:26.190969+0000","last_unstale":"2024-09-22T03:01:37.093896+0000","last_undegraded":"2024-09-22T03:01:07.306661+0000","last_fullsized":"2024-09-22T03:01:07.306661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1883,"log_dups_size":0,"ondisk_log_size":1883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123469824,"num_objects":1884,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1884,"num_write_kb":120576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:03.231131+0000","last_change":"2024-09-22T03:01:03.231788+0000","last_active":"2024-09-22T03:01:03.231131+0000","last_peered":"2024-09-22T03:01:03.231131+0000","last_clean":"2024-09-22T02:58:45.368663+0000","last_became_active":"2024-09-22T03:01:03.226859+0000","last_became_peered":"2024-09-22T03:01:03.226859+0000","last_unstale":"2024-09-22T03:01:03.231131+0000","last_undegraded":"2024-09-22T03:01:03.231131+0000","last_fullsized":"2024-09-22T03:01:03.231131+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'1892","reported_seq":1935,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-22T03:01:11.778507+0000","last_change":"2024-09-22T03:01:03.217552+0000","last_active":"2024-09-22T03:01:11.778507+0000","last_peered":"2024-09-22T03:01:11.778507+0000","last_clean":"2024-09-22T03:01:11.778507+0000","last_became_active":"2024-09-22T03:01:03.212561+0000","last_became_peered":"2024-09-22T03:01:03.212561+0000","last_unstale":"2024-09-22T03:01:11.778507+0000","last_undegraded":"2024-09-22T03:01:11.778507+0000","last_fullsized":"2024-09-22T03:01:11.778507+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1892,"log_dups_size":0,"ondisk_log_size":1892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124059648,"num_objects":1893,"num_object_clones":0,"num_object_copies":5679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1893,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.136294+0000","last_change":"2024-09-22T03:01:40.155884+0000","last_active":"2024-09-22T03:01:40.136294+0000","last_peered":"2024-09-22T03:01:40.136294+0000","last_clean":"2024-09-22T03:01:40.136294+0000","last_became_active":"2024-09-22T03:01:39.201350+0000","last_became_peered":"2024-09-22T03:01:39.201350+0000","last_unstale":"2024-09-22T03:01:40.136294+0000","last_undegraded":"2024-09-22T03:01:40.136294+0000","last_fullsized":"2024-09-22T03:01:40.136294+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"31'1843","reported_seq":1900,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.660073+0000","last_change":"2024-09-22T03:01:39.201120+0000","last_active":"2024-09-22T03:01:40.660073+0000","last_peered":"2024-09-22T03:01:40.660073+0000","last_clean":"2024-09-22T03:01:40.660073+0000","last_became_active":"2024-09-22T03:01:39.191673+0000","last_became_peered":"2024-09-22T03:01:39.191673+0000","last_unstale":"2024-09-22T03:01:40.660073+0000","last_undegraded":"2024-09-22T03:01:40.660073+0000","last_fullsized":"2024-09-22T03:01:40.660073+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1843,"log_dups_size":0,"ondisk_log_size":1843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117964800,"num_objects":1800,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1844,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.135447+0000","last_change":"2024-09-22T03:01:40.155556+0000","last_active":"2024-09-22T03:01:40.135447+0000","last_peered":"2024-09-22T03:01:40.135447+0000","last_clean":"2024-09-22T03:01:40.135447+0000","last_became_active":"2024-09-22T03:01:39.204853+0000","last_became_peered":"2024-09-22T03:01:39.204853+0000","last_unstale":"2024-09-22T03:01:40.135447+0000","last_undegraded":"2024-09-22T03:01:40.135447+0000","last_fullsized":"2024-09-22T03:01:40.135447+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1988165656,"num_objects":30338,"num_object_clones":0,"num_object_copies":91014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":30530,"num_write_kb":1947713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":371122176,"data_stored":5937889304,"data_compressed":281959780,"data_compressed_allocated":371118080,"data_compressed_original":5937889280,"omap_allocated":0,"internal_metadata":0},"log_size":30519,"ondisk_log_size":30519,"up":45,"acting":45,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":23,"acting":23,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673104,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":155344,"kb_used_data":102144,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94216496,"statfs":{"total":96636764160,"available":96477691904,"internally_reserved":0,"allocated":104595456,"data_stored":1669933723,"data_compressed":79292791,"data_compressed_allocated":104366080,"data_compressed_original":1669857280,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":26,"seq":111669149730,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126652,"kb_used_data":83820,"kb_used_omap":0,"kb_used_meta":42816,"kb_avail":94245188,"statfs":{"total":96636764160,"available":96507072512,"internally_reserved":0,"allocated":85831680,"data_stored":1369320635,"data_compressed":65020188,"data_compressed_allocated":85581824,"data_compressed_original":1369227264,"omap_allocated":0,"internal_metadata":43843584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":142356,"kb_used_data":84996,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94229484,"statfs":{"total":96636764160,"available":96490991616,"internally_reserved":0,"allocated":87035904,"data_stored":1388653736,"data_compressed":65938249,"data_compressed_allocated":86790144,"data_compressed_original":1388560384,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126620,"kb_used_data":92364,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":94245220,"statfs":{"total":96636764160,"available":96507105280,"internally_reserved":0,"allocated":94580736,"data_stored":1508729957,"data_compressed":71639538,"data_compressed_allocated":94294016,"data_compressed_original":1508622336,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86749184,"data_stored":1387986944,"data_compressed":65908290,"data_compressed_allocated":86749184,"data_compressed_original":1387986944,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":93523968,"data_stored":1496317976,"data_compressed":71052710,"data_compressed_allocated":93519872,"data_compressed_original":1496317952,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85540864,"data_stored":1368653824,"data_compressed":64990229,"data_compressed_allocated":85540864,"data_compressed_original":1368653824,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":105308160,"data_stored":1684930560,"data_compressed":80008551,"data_compressed_allocated":105308160,"data_compressed_original":1684930560,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:01:42.068 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:01:42.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:42.197+0000 7f2acbad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:42.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:42.207+0000 7f2acbad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:42.594 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:01:42.595 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg peering","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":188,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":32,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974098,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"stale+active+clean","count":4},{"state_name":"peering","count":4}],"num_pgs":25,"num_pools":3,"num_objects":30437,"data_bytes":1995047475,"bytes_used":564510720,"bytes_avail":385982545920,"bytes_total":386547056640,"inactive_pgs_ratio":0.15999999642372131,"write_bytes_sec":1408798,"read_op_per_sec":0,"write_op_per_sec":21},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:01:42.614 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-22T03:01:42.614 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:01:42.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:42.748+0000 7f1539e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:42.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:42.758+0000 7f1539e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:43.114 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:01:43.114 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:01:43.133 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-22T03:01:42.319007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1977090635,"num_objects":30164,"num_object_clones":0,"num_object_copies":90492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":30847,"num_write_kb":1949609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30797,"ondisk_log_size":30797,"up":71,"acting":71,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":553964,"kb_used_data":362152,"kb_used_omap":0,"kb_used_meta":191744,"kb_avail":376933396,"statfs":{"total":386547056640,"available":385979797504,"internally_reserved":0,"allocated":370843648,"data_stored":5916798142,"data_compressed":280948219,"data_compressed_allocated":369790976,"data_compressed_original":5916409856,"omap_allocated":0,"internal_metadata":196345856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17956840,"num_objects":-273,"num_object_clones":0,"num_object_copies":-819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":275,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.224232"},"pg_stats":[{"pgid":"3.f","version":"27'1941","reported_seq":1992,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:11.800110+0000","last_change":"2024-09-22T03:01:37.120882+0000","last_active":"2024-09-22T03:01:37.120882+0000","last_peered":"2024-09-22T03:01:11.800110+0000","last_clean":"2024-09-22T03:01:11.800110+0000","last_became_active":"2024-09-22T03:00:26.186244+0000","last_became_peered":"2024-09-22T03:00:26.186244+0000","last_unstale":"2024-09-22T03:01:11.800110+0000","last_undegraded":"2024-09-22T03:01:11.800110+0000","last_fullsized":"2024-09-22T03:01:11.800110+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1941,"log_dups_size":0,"ondisk_log_size":1941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127205376,"num_objects":1941,"num_object_clones":0,"num_object_copies":5823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1941,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"32'1946","reported_seq":2011,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.784602+0000","last_change":"2024-09-22T03:01:39.207406+0000","last_active":"2024-09-22T03:01:41.784602+0000","last_peered":"2024-09-22T03:01:41.784602+0000","last_clean":"2024-09-22T03:01:41.784602+0000","last_became_active":"2024-09-22T03:01:39.200120+0000","last_became_peered":"2024-09-22T03:01:39.200120+0000","last_unstale":"2024-09-22T03:01:41.784602+0000","last_undegraded":"2024-09-22T03:01:41.784602+0000","last_fullsized":"2024-09-22T03:01:41.784602+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1946,"log_dups_size":0,"ondisk_log_size":1946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123142144,"num_objects":1879,"num_object_clones":0,"num_object_copies":5637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1947,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'1947","reported_seq":2009,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.808541+0000","last_change":"2024-09-22T03:01:39.304739+0000","last_active":"2024-09-22T03:01:40.808541+0000","last_peered":"2024-09-22T03:01:40.808541+0000","last_clean":"2024-09-22T03:01:40.808541+0000","last_became_active":"2024-09-22T03:01:39.272458+0000","last_became_peered":"2024-09-22T03:01:39.272458+0000","last_unstale":"2024-09-22T03:01:40.808541+0000","last_undegraded":"2024-09-22T03:01:40.808541+0000","last_fullsized":"2024-09-22T03:01:40.808541+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1947,"log_dups_size":0,"ondisk_log_size":1947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124977176,"num_objects":1908,"num_object_clones":0,"num_object_copies":5724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1948,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'1847","reported_seq":1896,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:11.893913+0000","last_change":"2024-09-22T03:01:37.121638+0000","last_active":"2024-09-22T03:01:37.121638+0000","last_peered":"2024-09-22T03:01:11.893913+0000","last_clean":"2024-09-22T03:01:11.893913+0000","last_became_active":"2024-09-22T03:01:03.195863+0000","last_became_peered":"2024-09-22T03:01:03.195863+0000","last_unstale":"2024-09-22T03:01:11.893913+0000","last_undegraded":"2024-09-22T03:01:11.893913+0000","last_fullsized":"2024-09-22T03:01:11.893913+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1847,"log_dups_size":0,"ondisk_log_size":1847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121044992,"num_objects":1847,"num_object_clones":0,"num_object_copies":5541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1847,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'1904","reported_seq":1955,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:11.891523+0000","last_change":"2024-09-22T03:01:37.121582+0000","last_active":"2024-09-22T03:01:37.121582+0000","last_peered":"2024-09-22T03:01:11.891523+0000","last_clean":"2024-09-22T03:01:11.891523+0000","last_became_active":"2024-09-22T03:01:03.216120+0000","last_became_peered":"2024-09-22T03:01:03.216120+0000","last_unstale":"2024-09-22T03:01:11.891523+0000","last_undegraded":"2024-09-22T03:01:11.891523+0000","last_fullsized":"2024-09-22T03:01:11.891523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124780544,"num_objects":1904,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":121856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"27'1895","reported_seq":1949,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.118236+0000","last_change":"2024-09-22T03:01:03.193486+0000","last_active":"2024-09-22T03:01:37.118236+0000","last_peered":"2024-09-22T03:01:37.118236+0000","last_clean":"2024-09-22T03:01:37.118236+0000","last_became_active":"2024-09-22T03:01:03.189517+0000","last_became_peered":"2024-09-22T03:01:03.189517+0000","last_unstale":"2024-09-22T03:01:37.118236+0000","last_undegraded":"2024-09-22T03:01:37.118236+0000","last_fullsized":"2024-09-22T03:01:37.118236+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1895,"log_dups_size":0,"ondisk_log_size":1895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124190720,"num_objects":1895,"num_object_clones":0,"num_object_copies":5685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1895,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'1865","reported_seq":1917,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.123451+0000","last_change":"2024-09-22T03:01:03.363540+0000","last_active":"2024-09-22T03:01:37.123451+0000","last_peered":"2024-09-22T03:01:37.123451+0000","last_clean":"2024-09-22T03:01:37.123451+0000","last_became_active":"2024-09-22T03:01:03.209817+0000","last_became_peered":"2024-09-22T03:01:03.209817+0000","last_unstale":"2024-09-22T03:01:37.123451+0000","last_undegraded":"2024-09-22T03:01:37.123451+0000","last_fullsized":"2024-09-22T03:01:37.123451+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1865,"log_dups_size":0,"ondisk_log_size":1865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122224640,"num_objects":1865,"num_object_clones":0,"num_object_copies":5595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1865,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'1969","reported_seq":2031,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.819231+0000","last_change":"2024-09-22T03:01:39.209087+0000","last_active":"2024-09-22T03:01:40.819231+0000","last_peered":"2024-09-22T03:01:40.819231+0000","last_clean":"2024-09-22T03:01:40.819231+0000","last_became_active":"2024-09-22T03:01:39.196396+0000","last_became_peered":"2024-09-22T03:01:39.196396+0000","last_unstale":"2024-09-22T03:01:40.819231+0000","last_undegraded":"2024-09-22T03:01:40.819231+0000","last_fullsized":"2024-09-22T03:01:40.819231+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1969,"log_dups_size":0,"ondisk_log_size":1969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125304832,"num_objects":1912,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1970,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-22T03:01:03.143590+0000","last_change":"2024-09-22T03:01:01.102795+0000","last_active":"2024-09-22T03:01:01.102795+0000","last_peered":"2024-09-22T03:00:26.124572+0000","last_clean":"2024-09-22T03:00:26.124572+0000","last_became_active":"2024-09-22T02:58:45.367879+0000","last_became_peered":"2024-09-22T02:58:45.367879+0000","last_unstale":"2024-09-22T03:01:03.143590+0000","last_undegraded":"2024-09-22T03:01:03.143590+0000","last_fullsized":"2024-09-22T03:01:03.143590+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"31'1957","reported_seq":2013,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.832771+0000","last_change":"2024-09-22T03:01:39.198947+0000","last_active":"2024-09-22T03:01:40.832771+0000","last_peered":"2024-09-22T03:01:40.832771+0000","last_clean":"2024-09-22T03:01:40.832771+0000","last_became_active":"2024-09-22T03:01:39.186612+0000","last_became_peered":"2024-09-22T03:01:39.186612+0000","last_unstale":"2024-09-22T03:01:40.832771+0000","last_undegraded":"2024-09-22T03:01:40.832771+0000","last_fullsized":"2024-09-22T03:01:40.832771+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1957,"log_dups_size":0,"ondisk_log_size":1957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125042688,"num_objects":1908,"num_object_clones":0,"num_object_copies":5724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1958,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.109344+0000","last_change":"2024-09-22T03:01:37.119403+0000","last_active":"2024-09-22T03:01:37.109344+0000","last_peered":"2024-09-22T03:01:37.109344+0000","last_clean":"2024-09-22T03:01:37.109344+0000","last_became_active":"2024-09-22T03:01:03.184670+0000","last_became_peered":"2024-09-22T03:01:03.184670+0000","last_unstale":"2024-09-22T03:01:37.109344+0000","last_undegraded":"2024-09-22T03:01:37.109344+0000","last_fullsized":"2024-09-22T03:01:37.109344+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"32'2006","reported_seq":2070,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.805712+0000","last_change":"2024-09-22T03:01:39.280325+0000","last_active":"2024-09-22T03:01:41.805712+0000","last_peered":"2024-09-22T03:01:41.805712+0000","last_clean":"2024-09-22T03:01:41.805712+0000","last_became_active":"2024-09-22T03:01:39.204351+0000","last_became_peered":"2024-09-22T03:01:39.204351+0000","last_unstale":"2024-09-22T03:01:41.805712+0000","last_undegraded":"2024-09-22T03:01:41.805712+0000","last_fullsized":"2024-09-22T03:01:41.805712+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2006,"log_dups_size":0,"ondisk_log_size":2006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125239296,"num_objects":1911,"num_object_clones":0,"num_object_copies":5733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2007,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":75,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.132611+0000","last_change":"2024-09-22T03:01:40.118411+0000","last_active":"2024-09-22T03:01:41.132611+0000","last_peered":"2024-09-22T03:01:41.132611+0000","last_clean":"2024-09-22T03:01:41.132611+0000","last_became_active":"2024-09-22T03:01:39.208080+0000","last_became_peered":"2024-09-22T03:01:39.208080+0000","last_unstale":"2024-09-22T03:01:41.132611+0000","last_undegraded":"2024-09-22T03:01:41.132611+0000","last_fullsized":"2024-09-22T03:01:41.132611+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"27'1801","reported_seq":1858,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:06.858615+0000","last_change":"2024-09-22T03:01:03.365285+0000","last_active":"2024-09-22T03:01:06.858615+0000","last_peered":"2024-09-22T03:01:06.858615+0000","last_clean":"2024-09-22T03:01:06.858615+0000","last_became_active":"2024-09-22T03:01:03.180158+0000","last_became_peered":"2024-09-22T03:01:03.180158+0000","last_unstale":"2024-09-22T03:01:37.093892+0000","last_undegraded":"2024-09-22T03:01:06.858615+0000","last_fullsized":"2024-09-22T03:01:06.858615+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1801,"log_dups_size":0,"ondisk_log_size":1801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1802,"num_write_kb":115328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":58,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:03.148641+0000","last_change":"2024-09-22T03:01:01.124723+0000","last_active":"2024-09-22T03:01:03.148641+0000","last_peered":"2024-09-22T03:01:03.148641+0000","last_clean":"2024-09-22T03:01:03.148641+0000","last_became_active":"2024-09-22T03:00:26.191889+0000","last_became_peered":"2024-09-22T03:00:26.191889+0000","last_unstale":"2024-09-22T03:01:37.093893+0000","last_undegraded":"2024-09-22T03:01:03.148641+0000","last_fullsized":"2024-09-22T03:01:03.148641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":186,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-22T03:01:37.112908+0000","last_change":"2024-09-22T03:01:03.199665+0000","last_active":"2024-09-22T03:01:37.112908+0000","last_peered":"2024-09-22T03:01:37.112908+0000","last_clean":"2024-09-22T03:01:37.112908+0000","last_became_active":"2024-09-22T03:01:03.194697+0000","last_became_peered":"2024-09-22T03:01:03.194697+0000","last_unstale":"2024-09-22T03:01:37.112908+0000","last_undegraded":"2024-09-22T03:01:37.112908+0000","last_fullsized":"2024-09-22T03:01:37.112908+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'1944","reported_seq":1995,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:07.273651+0000","last_change":"2024-09-22T03:00:26.354667+0000","last_active":"2024-09-22T03:01:07.273651+0000","last_peered":"2024-09-22T03:01:07.273651+0000","last_clean":"2024-09-22T03:01:07.273651+0000","last_became_active":"2024-09-22T03:00:26.184456+0000","last_became_peered":"2024-09-22T03:00:26.184456+0000","last_unstale":"2024-09-22T03:01:37.093894+0000","last_undegraded":"2024-09-22T03:01:07.273651+0000","last_fullsized":"2024-09-22T03:01:07.273651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1944,"log_dups_size":0,"ondisk_log_size":1944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127467520,"num_objects":1945,"num_object_clones":0,"num_object_copies":5835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1945,"num_write_kb":124480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"32'2011","reported_seq":2071,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.833989+0000","last_change":"2024-09-22T03:01:39.223616+0000","last_active":"2024-09-22T03:01:41.833989+0000","last_peered":"2024-09-22T03:01:41.833989+0000","last_clean":"2024-09-22T03:01:41.833989+0000","last_became_active":"2024-09-22T03:01:39.218651+0000","last_became_peered":"2024-09-22T03:01:39.218651+0000","last_unstale":"2024-09-22T03:01:41.833989+0000","last_undegraded":"2024-09-22T03:01:41.833989+0000","last_fullsized":"2024-09-22T03:01:41.833989+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2011,"log_dups_size":0,"ondisk_log_size":2011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125304832,"num_objects":1912,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2012,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.136051+0000","last_change":"2024-09-22T03:01:37.133459+0000","last_active":"2024-09-22T03:01:40.136051+0000","last_peered":"2024-09-22T03:01:40.136051+0000","last_clean":"2024-09-22T03:01:40.136051+0000","last_became_active":"2024-09-22T03:01:03.184548+0000","last_became_peered":"2024-09-22T03:01:03.184548+0000","last_unstale":"2024-09-22T03:01:40.136051+0000","last_undegraded":"2024-09-22T03:01:40.136051+0000","last_fullsized":"2024-09-22T03:01:40.136051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"27'1883","reported_seq":1932,"reported_epoch":27,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:07.306661+0000","last_change":"2024-09-22T03:00:26.327795+0000","last_active":"2024-09-22T03:01:07.306661+0000","last_peered":"2024-09-22T03:01:07.306661+0000","last_clean":"2024-09-22T03:01:07.306661+0000","last_became_active":"2024-09-22T03:00:26.190969+0000","last_became_peered":"2024-09-22T03:00:26.190969+0000","last_unstale":"2024-09-22T03:01:37.093896+0000","last_undegraded":"2024-09-22T03:01:07.306661+0000","last_fullsized":"2024-09-22T03:01:07.306661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1883,"log_dups_size":0,"ondisk_log_size":1883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123469824,"num_objects":1884,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1884,"num_write_kb":120576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.129299+0000","last_change":"2024-09-22T03:01:40.122900+0000","last_active":"2024-09-22T03:01:41.129299+0000","last_peered":"2024-09-22T03:01:41.129299+0000","last_clean":"2024-09-22T03:01:41.129299+0000","last_became_active":"2024-09-22T03:01:39.184707+0000","last_became_peered":"2024-09-22T03:01:39.184707+0000","last_unstale":"2024-09-22T03:01:41.129299+0000","last_undegraded":"2024-09-22T03:01:41.129299+0000","last_fullsized":"2024-09-22T03:01:41.129299+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"32'1938","reported_seq":1997,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.792519+0000","last_change":"2024-09-22T03:01:39.195499+0000","last_active":"2024-09-22T03:01:41.792519+0000","last_peered":"2024-09-22T03:01:41.792519+0000","last_clean":"2024-09-22T03:01:41.792519+0000","last_became_active":"2024-09-22T03:01:39.181260+0000","last_became_peered":"2024-09-22T03:01:39.181260+0000","last_unstale":"2024-09-22T03:01:41.792519+0000","last_undegraded":"2024-09-22T03:01:41.792519+0000","last_fullsized":"2024-09-22T03:01:41.792519+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1938,"log_dups_size":0,"ondisk_log_size":1938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121044992,"num_objects":1847,"num_object_clones":0,"num_object_copies":5541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.136294+0000","last_change":"2024-09-22T03:01:40.155884+0000","last_active":"2024-09-22T03:01:40.136294+0000","last_peered":"2024-09-22T03:01:40.136294+0000","last_clean":"2024-09-22T03:01:40.136294+0000","last_became_active":"2024-09-22T03:01:39.201350+0000","last_became_peered":"2024-09-22T03:01:39.201350+0000","last_unstale":"2024-09-22T03:01:40.136294+0000","last_undegraded":"2024-09-22T03:01:40.136294+0000","last_fullsized":"2024-09-22T03:01:40.136294+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"31'1843","reported_seq":1900,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.660073+0000","last_change":"2024-09-22T03:01:39.201120+0000","last_active":"2024-09-22T03:01:40.660073+0000","last_peered":"2024-09-22T03:01:40.660073+0000","last_clean":"2024-09-22T03:01:40.660073+0000","last_became_active":"2024-09-22T03:01:39.191673+0000","last_became_peered":"2024-09-22T03:01:39.191673+0000","last_unstale":"2024-09-22T03:01:40.660073+0000","last_undegraded":"2024-09-22T03:01:40.660073+0000","last_fullsized":"2024-09-22T03:01:40.660073+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1843,"log_dups_size":0,"ondisk_log_size":1843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117964800,"num_objects":1800,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1844,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:01:40.135447+0000","last_change":"2024-09-22T03:01:40.155556+0000","last_active":"2024-09-22T03:01:40.135447+0000","last_peered":"2024-09-22T03:01:40.135447+0000","last_clean":"2024-09-22T03:01:40.135447+0000","last_became_active":"2024-09-22T03:01:39.204853+0000","last_became_peered":"2024-09-22T03:01:39.204853+0000","last_unstale":"2024-09-22T03:01:40.135447+0000","last_undegraded":"2024-09-22T03:01:40.135447+0000","last_fullsized":"2024-09-22T03:01:40.135447+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1976500248,"num_objects":30160,"num_object_clones":0,"num_object_copies":90480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":30708,"num_write_kb":1947713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":369266688,"data_stored":5908201496,"data_compressed":280550583,"data_compressed_allocated":369262592,"data_compressed_original":5908201472,"omap_allocated":0,"internal_metadata":0},"log_size":30697,"ondisk_log_size":30697,"up":45,"acting":45,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":23,"acting":23,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673104,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":155344,"kb_used_data":102144,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94216496,"statfs":{"total":96636764160,"available":96477691904,"internally_reserved":0,"allocated":104595456,"data_stored":1669933723,"data_compressed":79292791,"data_compressed_allocated":104366080,"data_compressed_original":1669857280,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":26,"seq":111669149735,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":129644,"kb_used_data":82648,"kb_used_omap":0,"kb_used_meta":46976,"kb_avail":94242196,"statfs":{"total":96636764160,"available":96504008704,"internally_reserved":0,"allocated":84631552,"data_stored":1349480726,"data_compressed":64077641,"data_compressed_allocated":84340736,"data_compressed_original":1349369856,"omap_allocated":0,"internal_metadata":48103424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":142356,"kb_used_data":84996,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94229484,"statfs":{"total":96636764160,"available":96490991616,"internally_reserved":0,"allocated":87035904,"data_stored":1388653736,"data_compressed":65938249,"data_compressed_allocated":86790144,"data_compressed_original":1388560384,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126620,"kb_used_data":92364,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":94245220,"statfs":{"total":96636764160,"available":96507105280,"internally_reserved":0,"allocated":94580736,"data_stored":1508729957,"data_compressed":71639538,"data_compressed_allocated":94294016,"data_compressed_original":1508622336,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86749184,"data_stored":1387986944,"data_compressed":65908290,"data_compressed_allocated":86749184,"data_compressed_original":1387986944,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":93523968,"data_stored":1496317976,"data_compressed":71052710,"data_compressed_allocated":93519872,"data_compressed_original":1496317952,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83685376,"data_stored":1338966016,"data_compressed":63581032,"data_compressed_allocated":83685376,"data_compressed_original":1338966016,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":105308160,"data_stored":1684930560,"data_compressed":80008551,"data_compressed_allocated":105308160,"data_compressed_original":1684930560,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:01:43.818 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:01:43.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:43.933+0000 7fc1fc397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:43.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:43.939+0000 7fc1fc397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:01:46.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:46.277+0000 7f510f997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:46.288+0000 7f510f997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.372 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:01:46.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:46.517+0000 7fe206063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:01:46.525+0000 7fe206063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.571 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:01:46.571 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:01:46.586 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-22T03:01:46.319755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1939800651,"num_objects":29595,"num_object_clones":0,"num_object_copies":88785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":31960,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31905,"ondisk_log_size":31905,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":533064,"kb_used_data":358528,"kb_used_omap":0,"kb_used_meta":174464,"kb_avail":376954296,"statfs":{"total":386547056640,"available":386001199104,"internally_reserved":0,"allocated":367132672,"data_stored":5855887981,"data_compressed":278056113,"data_compressed_allocated":365981696,"data_compressed_original":5855461376,"omap_allocated":0,"internal_metadata":178651136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-55246824,"num_objects":-842,"num_object_clones":0,"num_object_copies":-2526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1388,"num_write_kb":17409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.222519"},"pg_stats":[{"pgid":"3.f","version":"32'1987","reported_seq":2054,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:42.284163+0000","last_change":"2024-09-22T03:01:39.185084+0000","last_active":"2024-09-22T03:01:42.284163+0000","last_peered":"2024-09-22T03:01:42.284163+0000","last_clean":"2024-09-22T03:01:42.284163+0000","last_became_active":"2024-09-22T03:01:39.181114+0000","last_became_peered":"2024-09-22T03:01:39.181114+0000","last_unstale":"2024-09-22T03:01:42.284163+0000","last_undegraded":"2024-09-22T03:01:42.284163+0000","last_fullsized":"2024-09-22T03:01:42.284163+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1987,"log_dups_size":0,"ondisk_log_size":1987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124125184,"num_objects":1894,"num_object_clones":0,"num_object_copies":5682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1988,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"32'1946","reported_seq":2011,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.784602+0000","last_change":"2024-09-22T03:01:39.207406+0000","last_active":"2024-09-22T03:01:41.784602+0000","last_peered":"2024-09-22T03:01:41.784602+0000","last_clean":"2024-09-22T03:01:41.784602+0000","last_became_active":"2024-09-22T03:01:39.200120+0000","last_became_peered":"2024-09-22T03:01:39.200120+0000","last_unstale":"2024-09-22T03:01:41.784602+0000","last_undegraded":"2024-09-22T03:01:41.784602+0000","last_fullsized":"2024-09-22T03:01:41.784602+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1946,"log_dups_size":0,"ondisk_log_size":1946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123142144,"num_objects":1879,"num_object_clones":0,"num_object_copies":5637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1947,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"32'2033","reported_seq":2098,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.850647+0000","last_change":"2024-09-22T03:01:39.304739+0000","last_active":"2024-09-22T03:01:45.850647+0000","last_peered":"2024-09-22T03:01:45.850647+0000","last_clean":"2024-09-22T03:01:45.850647+0000","last_became_active":"2024-09-22T03:01:39.272458+0000","last_became_peered":"2024-09-22T03:01:39.272458+0000","last_unstale":"2024-09-22T03:01:45.850647+0000","last_undegraded":"2024-09-22T03:01:45.850647+0000","last_fullsized":"2024-09-22T03:01:45.850647+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2033,"log_dups_size":0,"ondisk_log_size":2033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119341080,"num_objects":1822,"num_object_clones":0,"num_object_copies":5466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2034,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"32'1892","reported_seq":1962,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:42.129429+0000","last_change":"2024-09-22T03:01:39.254261+0000","last_active":"2024-09-22T03:01:42.129429+0000","last_peered":"2024-09-22T03:01:42.129429+0000","last_clean":"2024-09-22T03:01:42.129429+0000","last_became_active":"2024-09-22T03:01:39.165796+0000","last_became_peered":"2024-09-22T03:01:39.165796+0000","last_unstale":"2024-09-22T03:01:42.129429+0000","last_undegraded":"2024-09-22T03:01:42.129429+0000","last_fullsized":"2024-09-22T03:01:42.129429+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1892,"log_dups_size":0,"ondisk_log_size":1892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118030336,"num_objects":1801,"num_object_clones":0,"num_object_copies":5403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1893,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"32'1976","reported_seq":2046,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:43.305645+0000","last_change":"2024-09-22T03:01:39.285980+0000","last_active":"2024-09-22T03:01:43.305645+0000","last_peered":"2024-09-22T03:01:43.305645+0000","last_clean":"2024-09-22T03:01:43.305645+0000","last_became_active":"2024-09-22T03:01:39.191952+0000","last_became_peered":"2024-09-22T03:01:39.191952+0000","last_unstale":"2024-09-22T03:01:43.305645+0000","last_undegraded":"2024-09-22T03:01:43.305645+0000","last_fullsized":"2024-09-22T03:01:43.305645+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1976,"log_dups_size":0,"ondisk_log_size":1976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120127488,"num_objects":1833,"num_object_clones":0,"num_object_copies":5499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1977,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"32'1936","reported_seq":2003,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:42.246937+0000","last_change":"2024-09-22T03:01:03.193486+0000","last_active":"2024-09-22T03:01:42.246937+0000","last_peered":"2024-09-22T03:01:42.246937+0000","last_clean":"2024-09-22T03:01:42.246937+0000","last_became_active":"2024-09-22T03:01:03.189517+0000","last_became_peered":"2024-09-22T03:01:03.189517+0000","last_unstale":"2024-09-22T03:01:42.246937+0000","last_undegraded":"2024-09-22T03:01:42.246937+0000","last_fullsized":"2024-09-22T03:01:42.246937+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1936,"log_dups_size":0,"ondisk_log_size":1936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121438208,"num_objects":1853,"num_object_clones":0,"num_object_copies":5559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1937,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"32'1919","reported_seq":1984,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:42.292495+0000","last_change":"2024-09-22T03:01:03.363540+0000","last_active":"2024-09-22T03:01:42.292495+0000","last_peered":"2024-09-22T03:01:42.292495+0000","last_clean":"2024-09-22T03:01:42.292495+0000","last_became_active":"2024-09-22T03:01:03.209817+0000","last_became_peered":"2024-09-22T03:01:03.209817+0000","last_unstale":"2024-09-22T03:01:42.292495+0000","last_undegraded":"2024-09-22T03:01:42.292495+0000","last_fullsized":"2024-09-22T03:01:42.292495+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1919,"log_dups_size":0,"ondisk_log_size":1919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118620160,"num_objects":1810,"num_object_clones":0,"num_object_copies":5430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1920,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"32'2073","reported_seq":2138,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.815245+0000","last_change":"2024-09-22T03:01:39.209087+0000","last_active":"2024-09-22T03:01:45.815245+0000","last_peered":"2024-09-22T03:01:45.815245+0000","last_clean":"2024-09-22T03:01:45.815245+0000","last_became_active":"2024-09-22T03:01:39.196396+0000","last_became_peered":"2024-09-22T03:01:39.196396+0000","last_unstale":"2024-09-22T03:01:45.815245+0000","last_undegraded":"2024-09-22T03:01:45.815245+0000","last_fullsized":"2024-09-22T03:01:45.815245+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118489088,"num_objects":1808,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2074,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.158747+0000","last_change":"2024-09-22T03:01:40.146115+0000","last_active":"2024-09-22T03:01:41.158747+0000","last_peered":"2024-09-22T03:01:41.158747+0000","last_clean":"2024-09-22T03:01:41.158747+0000","last_became_active":"2024-09-22T03:01:39.198270+0000","last_became_peered":"2024-09-22T03:01:39.198270+0000","last_unstale":"2024-09-22T03:01:41.158747+0000","last_undegraded":"2024-09-22T03:01:41.158747+0000","last_fullsized":"2024-09-22T03:01:41.158747+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"32'2057","reported_seq":2116,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.825848+0000","last_change":"2024-09-22T03:01:39.198947+0000","last_active":"2024-09-22T03:01:45.825848+0000","last_peered":"2024-09-22T03:01:45.825848+0000","last_clean":"2024-09-22T03:01:45.825848+0000","last_became_active":"2024-09-22T03:01:39.186612+0000","last_became_peered":"2024-09-22T03:01:39.186612+0000","last_unstale":"2024-09-22T03:01:45.825848+0000","last_undegraded":"2024-09-22T03:01:45.825848+0000","last_fullsized":"2024-09-22T03:01:45.825848+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118489088,"num_objects":1808,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2058,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":73,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.111538+0000","last_change":"2024-09-22T03:01:37.119403+0000","last_active":"2024-09-22T03:01:41.111538+0000","last_peered":"2024-09-22T03:01:41.111538+0000","last_clean":"2024-09-22T03:01:41.111538+0000","last_became_active":"2024-09-22T03:01:03.184670+0000","last_became_peered":"2024-09-22T03:01:03.184670+0000","last_unstale":"2024-09-22T03:01:41.111538+0000","last_undegraded":"2024-09-22T03:01:41.111538+0000","last_fullsized":"2024-09-22T03:01:41.111538+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"32'2006","reported_seq":2070,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.805712+0000","last_change":"2024-09-22T03:01:39.280325+0000","last_active":"2024-09-22T03:01:41.805712+0000","last_peered":"2024-09-22T03:01:41.805712+0000","last_clean":"2024-09-22T03:01:41.805712+0000","last_became_active":"2024-09-22T03:01:39.204351+0000","last_became_peered":"2024-09-22T03:01:39.204351+0000","last_unstale":"2024-09-22T03:01:41.805712+0000","last_undegraded":"2024-09-22T03:01:41.805712+0000","last_fullsized":"2024-09-22T03:01:41.805712+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2006,"log_dups_size":0,"ondisk_log_size":2006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125239296,"num_objects":1911,"num_object_clones":0,"num_object_copies":5733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2007,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":75,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.132611+0000","last_change":"2024-09-22T03:01:40.118411+0000","last_active":"2024-09-22T03:01:41.132611+0000","last_peered":"2024-09-22T03:01:41.132611+0000","last_clean":"2024-09-22T03:01:41.132611+0000","last_became_active":"2024-09-22T03:01:39.208080+0000","last_became_peered":"2024-09-22T03:01:39.208080+0000","last_unstale":"2024-09-22T03:01:41.132611+0000","last_undegraded":"2024-09-22T03:01:41.132611+0000","last_fullsized":"2024-09-22T03:01:41.132611+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"32'1942","reported_seq":2014,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:43.277318+0000","last_change":"2024-09-22T03:01:39.230640+0000","last_active":"2024-09-22T03:01:43.277318+0000","last_peered":"2024-09-22T03:01:43.277318+0000","last_clean":"2024-09-22T03:01:43.277318+0000","last_became_active":"2024-09-22T03:01:39.217644+0000","last_became_peered":"2024-09-22T03:01:39.217644+0000","last_unstale":"2024-09-22T03:01:43.277318+0000","last_undegraded":"2024-09-22T03:01:43.277318+0000","last_fullsized":"2024-09-22T03:01:43.277318+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1942,"log_dups_size":0,"ondisk_log_size":1942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119734272,"num_objects":1827,"num_object_clones":0,"num_object_copies":5481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1943,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":64,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.159282+0000","last_change":"2024-09-22T03:01:40.146323+0000","last_active":"2024-09-22T03:01:41.159282+0000","last_peered":"2024-09-22T03:01:41.159282+0000","last_clean":"2024-09-22T03:01:41.159282+0000","last_became_active":"2024-09-22T03:01:39.187168+0000","last_became_peered":"2024-09-22T03:01:39.187168+0000","last_unstale":"2024-09-22T03:01:41.159282+0000","last_undegraded":"2024-09-22T03:01:41.159282+0000","last_fullsized":"2024-09-22T03:01:41.159282+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":198,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.114712+0000","last_change":"2024-09-22T03:01:03.199665+0000","last_active":"2024-09-22T03:01:41.114712+0000","last_peered":"2024-09-22T03:01:41.114712+0000","last_clean":"2024-09-22T03:01:41.114712+0000","last_became_active":"2024-09-22T03:01:03.194697+0000","last_became_peered":"2024-09-22T03:01:03.194697+0000","last_unstale":"2024-09-22T03:01:41.114712+0000","last_undegraded":"2024-09-22T03:01:41.114712+0000","last_fullsized":"2024-09-22T03:01:41.114712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"32'2100","reported_seq":2166,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:43.252495+0000","last_change":"2024-09-22T03:01:39.198416+0000","last_active":"2024-09-22T03:01:43.252495+0000","last_peered":"2024-09-22T03:01:43.252495+0000","last_clean":"2024-09-22T03:01:43.252495+0000","last_became_active":"2024-09-22T03:01:39.188561+0000","last_became_peered":"2024-09-22T03:01:39.188561+0000","last_unstale":"2024-09-22T03:01:43.252495+0000","last_undegraded":"2024-09-22T03:01:43.252495+0000","last_fullsized":"2024-09-22T03:01:43.252495+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128253952,"num_objects":1957,"num_object_clones":0,"num_object_copies":5871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"32'2011","reported_seq":2071,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.833989+0000","last_change":"2024-09-22T03:01:39.223616+0000","last_active":"2024-09-22T03:01:41.833989+0000","last_peered":"2024-09-22T03:01:41.833989+0000","last_clean":"2024-09-22T03:01:41.833989+0000","last_became_active":"2024-09-22T03:01:39.218651+0000","last_became_peered":"2024-09-22T03:01:39.218651+0000","last_unstale":"2024-09-22T03:01:41.833989+0000","last_undegraded":"2024-09-22T03:01:41.833989+0000","last_fullsized":"2024-09-22T03:01:41.833989+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2011,"log_dups_size":0,"ondisk_log_size":2011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125304832,"num_objects":1912,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2012,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.116616+0000","last_change":"2024-09-22T03:01:37.133459+0000","last_active":"2024-09-22T03:01:41.116616+0000","last_peered":"2024-09-22T03:01:41.116616+0000","last_clean":"2024-09-22T03:01:41.116616+0000","last_became_active":"2024-09-22T03:01:03.184548+0000","last_became_peered":"2024-09-22T03:01:03.184548+0000","last_unstale":"2024-09-22T03:01:41.116616+0000","last_undegraded":"2024-09-22T03:01:41.116616+0000","last_fullsized":"2024-09-22T03:01:41.116616+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"32'2056","reported_seq":2120,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:43.266064+0000","last_change":"2024-09-22T03:01:39.202507+0000","last_active":"2024-09-22T03:01:43.266064+0000","last_peered":"2024-09-22T03:01:43.266064+0000","last_clean":"2024-09-22T03:01:43.266064+0000","last_became_active":"2024-09-22T03:01:39.191156+0000","last_became_peered":"2024-09-22T03:01:39.191156+0000","last_unstale":"2024-09-22T03:01:43.266064+0000","last_undegraded":"2024-09-22T03:01:43.266064+0000","last_fullsized":"2024-09-22T03:01:43.266064+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2056,"log_dups_size":0,"ondisk_log_size":2056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125763584,"num_objects":1919,"num_object_clones":0,"num_object_copies":5757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2057,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.129299+0000","last_change":"2024-09-22T03:01:40.122900+0000","last_active":"2024-09-22T03:01:41.129299+0000","last_peered":"2024-09-22T03:01:41.129299+0000","last_clean":"2024-09-22T03:01:41.129299+0000","last_became_active":"2024-09-22T03:01:39.184707+0000","last_became_peered":"2024-09-22T03:01:39.184707+0000","last_unstale":"2024-09-22T03:01:41.129299+0000","last_undegraded":"2024-09-22T03:01:41.129299+0000","last_fullsized":"2024-09-22T03:01:41.129299+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"32'1938","reported_seq":1997,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.792519+0000","last_change":"2024-09-22T03:01:39.195499+0000","last_active":"2024-09-22T03:01:41.792519+0000","last_peered":"2024-09-22T03:01:41.792519+0000","last_clean":"2024-09-22T03:01:41.792519+0000","last_became_active":"2024-09-22T03:01:39.181260+0000","last_became_peered":"2024-09-22T03:01:39.181260+0000","last_unstale":"2024-09-22T03:01:41.792519+0000","last_undegraded":"2024-09-22T03:01:41.792519+0000","last_fullsized":"2024-09-22T03:01:41.792519+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1938,"log_dups_size":0,"ondisk_log_size":1938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121044992,"num_objects":1847,"num_object_clones":0,"num_object_copies":5541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.117172+0000","last_change":"2024-09-22T03:01:40.155884+0000","last_active":"2024-09-22T03:01:41.117172+0000","last_peered":"2024-09-22T03:01:41.117172+0000","last_clean":"2024-09-22T03:01:41.117172+0000","last_became_active":"2024-09-22T03:01:39.201350+0000","last_became_peered":"2024-09-22T03:01:39.201350+0000","last_unstale":"2024-09-22T03:01:41.117172+0000","last_undegraded":"2024-09-22T03:01:41.117172+0000","last_fullsized":"2024-09-22T03:01:41.117172+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"32'1933","reported_seq":1993,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.861809+0000","last_change":"2024-09-22T03:01:39.201120+0000","last_active":"2024-09-22T03:01:45.861809+0000","last_peered":"2024-09-22T03:01:45.861809+0000","last_clean":"2024-09-22T03:01:45.861809+0000","last_became_active":"2024-09-22T03:01:39.191673+0000","last_became_peered":"2024-09-22T03:01:39.191673+0000","last_unstale":"2024-09-22T03:01:45.861809+0000","last_undegraded":"2024-09-22T03:01:45.861809+0000","last_fullsized":"2024-09-22T03:01:45.861809+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1933,"log_dups_size":0,"ondisk_log_size":1933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112066560,"num_objects":1710,"num_object_clones":0,"num_object_copies":5130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1934,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:01:41.118171+0000","last_change":"2024-09-22T03:01:40.155556+0000","last_active":"2024-09-22T03:01:41.118171+0000","last_peered":"2024-09-22T03:01:41.118171+0000","last_clean":"2024-09-22T03:01:41.118171+0000","last_became_active":"2024-09-22T03:01:39.204853+0000","last_became_peered":"2024-09-22T03:01:39.204853+0000","last_unstale":"2024-09-22T03:01:41.118171+0000","last_undegraded":"2024-09-22T03:01:41.118171+0000","last_fullsized":"2024-09-22T03:01:41.118171+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1939210264,"num_objects":29591,"num_object_clones":0,"num_object_copies":88773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":31821,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":363610112,"data_stored":5817565256,"data_compressed":276248732,"data_compressed_allocated":363597824,"data_compressed_original":5817565184,"omap_allocated":0,"internal_metadata":0},"log_size":31805,"ondisk_log_size":31805,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":29,"seq":124554051589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":136260,"kb_used_data":104500,"kb_used_omap":0,"kb_used_meta":31744,"kb_avail":94235580,"statfs":{"total":96636764160,"available":96497233920,"internally_reserved":0,"allocated":107008000,"data_stored":1707831054,"data_compressed":81092495,"data_compressed_allocated":106733568,"data_compressed_original":1707737088,"omap_allocated":0,"internal_metadata":32505856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":26,"seq":111669149735,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":129644,"kb_used_data":82648,"kb_used_omap":0,"kb_used_meta":46976,"kb_avail":94242196,"statfs":{"total":96636764160,"available":96504008704,"internally_reserved":0,"allocated":84631552,"data_stored":1349480726,"data_compressed":64077641,"data_compressed_allocated":84340736,"data_compressed_original":1349369856,"omap_allocated":0,"internal_metadata":48103424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673139,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":140940,"kb_used_data":83580,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94230900,"statfs":{"total":96636764160,"available":96492441600,"internally_reserved":0,"allocated":85585920,"data_stored":1364750619,"data_compressed":64802804,"data_compressed_allocated":85295104,"data_compressed_original":1364639744,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126220,"kb_used_data":87800,"kb_used_omap":0,"kb_used_meta":38400,"kb_avail":94245620,"statfs":{"total":96636764160,"available":96507514880,"internally_reserved":0,"allocated":89907200,"data_stored":1433825582,"data_compressed":68083173,"data_compressed_allocated":89612288,"data_compressed_original":1433714688,"omap_allocated":0,"internal_metadata":39321600},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":84561920,"data_stored":1352925208,"data_compressed":64243975,"data_compressed_allocated":84557824,"data_compressed_original":1352925184,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88866816,"data_stored":1421803544,"data_compressed":67514838,"data_compressed_allocated":88862720,"data_compressed_original":1421803520,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83685376,"data_stored":1338966016,"data_compressed":63581032,"data_compressed_allocated":83685376,"data_compressed_original":1338966016,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496000,"data_stored":1703870488,"data_compressed":80908887,"data_compressed_allocated":106491904,"data_compressed_original":1703870464,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:01:46.590 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-22T03:01:46.590 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:01:46.590 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3] 2024-09-22T03:01:46.591 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:01:46.591 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:01:47.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:01:52.692 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:01:52.953 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~6s 2024-09-22T03:01:57.694 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3] 2024-09-22T03:01:57.694 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:01:57.694 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-22T03:01:57.694 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:01:57.694 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:01:57.697 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:01:57.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:01:57.807 INFO:teuthology.orchestra.run.smithi081.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:01:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:01:57.811 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:01:58.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:02:02.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:02:02.907 INFO:teuthology.orchestra.run.smithi081.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:02:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:02:02.910 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:02:07.911 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:02:08.007 INFO:teuthology.orchestra.run.smithi081.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:02:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:02:08.010 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:02:13.011 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:02:13.094 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:02:13.143 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: "ops_in_flight": [ 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "pg_advance_map", 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 33554446, 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: "PGAdvanceMap": { 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "1.4", 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 27, 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 33, 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: "do_init": false 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:02:13.144 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "pg_advance_map", 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 33554447, 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: "detail": { 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: "PGAdvanceMap": { 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: "pgid": "2.0", 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: "from": 27, 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: "to": 33, 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: "do_init": false 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-09-22T03:02:13.145 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-22T03:02:13.198 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:02:13.198 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:02:15.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:02:15.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:15.326+0000 7fd0f9957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:15.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:15.334+0000 7fd0f9957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:17.698 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:02:17.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:17.843+0000 7f8bcd917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:17.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:17.851+0000 7f8bcd917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:18.157 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-09-22T03:02:18.157 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:02:18.157 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:02:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:18.284+0000 7fb60d117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:18.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:18.292+0000 7fb60d117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:18.527 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:02:18.528 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:02:18.541 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-22T03:02:18.326847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1857421899,"num_objects":28338,"num_object_clones":0,"num_object_copies":85014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":33217,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33169,"ondisk_log_size":33169,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":501976,"kb_used_data":344084,"kb_used_omap":0,"kb_used_meta":157824,"kb_avail":376985384,"statfs":{"total":386547056640,"available":386033033216,"internally_reserved":0,"allocated":352342016,"data_stored":5618345443,"data_compressed":266775823,"data_compressed_allocated":351133696,"data_compressed_original":5617893376,"omap_allocated":0,"internal_metadata":161611776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20250624,"num_objects":-309,"num_object_clones":0,"num_object_copies":-927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":309,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001222"},"pg_stats":[{"pgid":"3.f","version":"35'2109","reported_seq":2192,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:17.121501+0000","last_change":"2024-09-22T03:02:15.329405+0000","last_active":"2024-09-22T03:02:17.121501+0000","last_peered":"2024-09-22T03:02:17.121501+0000","last_clean":"2024-09-22T03:02:17.121501+0000","last_became_active":"2024-09-22T03:02:15.205488+0000","last_became_peered":"2024-09-22T03:02:15.205488+0000","last_unstale":"2024-09-22T03:02:17.121501+0000","last_undegraded":"2024-09-22T03:02:17.121501+0000","last_fullsized":"2024-09-22T03:02:17.121501+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2109,"log_dups_size":0,"ondisk_log_size":2109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116129792,"num_objects":1772,"num_object_clones":0,"num_object_copies":5316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2110,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"35'2069","reported_seq":2154,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:16.708967+0000","last_change":"2024-09-22T03:02:15.425881+0000","last_active":"2024-09-22T03:02:16.708967+0000","last_peered":"2024-09-22T03:02:16.708967+0000","last_clean":"2024-09-22T03:02:16.708967+0000","last_became_active":"2024-09-22T03:02:15.201857+0000","last_became_peered":"2024-09-22T03:02:15.201857+0000","last_unstale":"2024-09-22T03:02:16.708967+0000","last_undegraded":"2024-09-22T03:02:16.708967+0000","last_fullsized":"2024-09-22T03:02:16.708967+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2069,"log_dups_size":0,"ondisk_log_size":2069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115081216,"num_objects":1756,"num_object_clones":0,"num_object_copies":5268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2070,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"32'2046","reported_seq":2111,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-22T03:01:46.351806+0000","last_change":"2024-09-22T03:02:13.114982+0000","last_active":"2024-09-22T03:02:13.114982+0000","last_peered":"2024-09-22T03:01:46.351806+0000","last_clean":"2024-09-22T03:01:46.351806+0000","last_became_active":"2024-09-22T03:01:39.272458+0000","last_became_peered":"2024-09-22T03:01:39.272458+0000","last_unstale":"2024-09-22T03:01:46.351806+0000","last_undegraded":"2024-09-22T03:01:46.351806+0000","last_fullsized":"2024-09-22T03:01:46.351806+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2046,"log_dups_size":0,"ondisk_log_size":2046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118554648,"num_objects":1810,"num_object_clones":0,"num_object_copies":5430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2046,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"35'2012","reported_seq":2091,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:17.298501+0000","last_change":"2024-09-22T03:01:39.254261+0000","last_active":"2024-09-22T03:02:17.298501+0000","last_peered":"2024-09-22T03:02:17.298501+0000","last_clean":"2024-09-22T03:02:17.298501+0000","last_became_active":"2024-09-22T03:01:39.165796+0000","last_became_peered":"2024-09-22T03:01:39.165796+0000","last_unstale":"2024-09-22T03:02:17.298501+0000","last_undegraded":"2024-09-22T03:02:17.298501+0000","last_fullsized":"2024-09-22T03:02:17.298501+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2012,"log_dups_size":0,"ondisk_log_size":2012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110166016,"num_objects":1681,"num_object_clones":0,"num_object_copies":5043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2013,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"32'2051","reported_seq":2123,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-22T03:02:13.112060+0000","last_change":"2024-09-22T03:01:39.285980+0000","last_active":"2024-09-22T03:02:13.112060+0000","last_peered":"2024-09-22T03:02:13.112060+0000","last_clean":"2024-09-22T03:02:13.112060+0000","last_became_active":"2024-09-22T03:01:39.191952+0000","last_became_peered":"2024-09-22T03:01:39.191952+0000","last_unstale":"2024-09-22T03:02:13.112060+0000","last_undegraded":"2024-09-22T03:02:13.112060+0000","last_fullsized":"2024-09-22T03:02:13.112060+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115277824,"num_objects":1759,"num_object_clones":0,"num_object_copies":5277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2051,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"35'2052","reported_seq":2129,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:17.253536+0000","last_change":"2024-09-22T03:02:15.196410+0000","last_active":"2024-09-22T03:02:17.253536+0000","last_peered":"2024-09-22T03:02:17.253536+0000","last_clean":"2024-09-22T03:02:17.253536+0000","last_became_active":"2024-09-22T03:02:15.193360+0000","last_became_peered":"2024-09-22T03:02:15.193360+0000","last_unstale":"2024-09-22T03:02:17.253536+0000","last_undegraded":"2024-09-22T03:02:17.253536+0000","last_fullsized":"2024-09-22T03:02:17.253536+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113836032,"num_objects":1737,"num_object_clones":0,"num_object_copies":5211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2053,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"35'2040","reported_seq":2115,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:17.234556+0000","last_change":"2024-09-22T03:02:15.183663+0000","last_active":"2024-09-22T03:02:17.234556+0000","last_peered":"2024-09-22T03:02:17.234556+0000","last_clean":"2024-09-22T03:02:17.234556+0000","last_became_active":"2024-09-22T03:02:15.175035+0000","last_became_peered":"2024-09-22T03:02:15.175035+0000","last_unstale":"2024-09-22T03:02:17.234556+0000","last_undegraded":"2024-09-22T03:02:17.234556+0000","last_fullsized":"2024-09-22T03:02:17.234556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110690304,"num_objects":1689,"num_object_clones":0,"num_object_copies":5067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2041,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"32'2083","reported_seq":2148,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-22T03:01:46.573388+0000","last_change":"2024-09-22T03:02:13.110690+0000","last_active":"2024-09-22T03:02:13.110690+0000","last_peered":"2024-09-22T03:01:46.573388+0000","last_clean":"2024-09-22T03:01:46.573388+0000","last_became_active":"2024-09-22T03:01:39.196396+0000","last_became_peered":"2024-09-22T03:01:39.196396+0000","last_unstale":"2024-09-22T03:01:46.573388+0000","last_undegraded":"2024-09-22T03:01:46.573388+0000","last_fullsized":"2024-09-22T03:01:46.573388+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117899264,"num_objects":1799,"num_object_clones":0,"num_object_copies":5397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2083,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-22T03:02:13.115219+0000","last_change":"2024-09-22T03:01:40.146115+0000","last_active":"2024-09-22T03:02:13.115219+0000","last_peered":"2024-09-22T03:02:13.115219+0000","last_clean":"2024-09-22T03:02:13.115219+0000","last_became_active":"2024-09-22T03:01:39.198270+0000","last_became_peered":"2024-09-22T03:01:39.198270+0000","last_unstale":"2024-09-22T03:02:13.115219+0000","last_undegraded":"2024-09-22T03:02:13.115219+0000","last_fullsized":"2024-09-22T03:02:13.115219+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"32'2057","reported_seq":2116,"reported_epoch":32,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:45.825848+0000","last_change":"2024-09-22T03:01:39.198947+0000","last_active":"2024-09-22T03:01:45.825848+0000","last_peered":"2024-09-22T03:01:45.825848+0000","last_clean":"2024-09-22T03:01:45.825848+0000","last_became_active":"2024-09-22T03:01:39.186612+0000","last_became_peered":"2024-09-22T03:01:39.186612+0000","last_unstale":"2024-09-22T03:02:13.091039+0000","last_undegraded":"2024-09-22T03:01:45.825848+0000","last_fullsized":"2024-09-22T03:01:45.825848+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118489088,"num_objects":1808,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2058,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":83,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.164068+0000","last_change":"2024-09-22T03:02:15.164748+0000","last_active":"2024-09-22T03:02:15.164068+0000","last_peered":"2024-09-22T03:02:15.164068+0000","last_clean":"2024-09-22T03:01:41.112371+0000","last_became_active":"2024-09-22T03:02:15.159837+0000","last_became_peered":"2024-09-22T03:02:15.159837+0000","last_unstale":"2024-09-22T03:02:15.164068+0000","last_undegraded":"2024-09-22T03:02:15.164068+0000","last_fullsized":"2024-09-22T03:02:15.164068+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"35'2120","reported_seq":2193,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:16.733274+0000","last_change":"2024-09-22T03:01:39.280325+0000","last_active":"2024-09-22T03:02:16.733274+0000","last_peered":"2024-09-22T03:02:16.733274+0000","last_clean":"2024-09-22T03:02:16.733274+0000","last_became_active":"2024-09-22T03:01:39.204351+0000","last_became_peered":"2024-09-22T03:01:39.204351+0000","last_unstale":"2024-09-22T03:02:16.733274+0000","last_undegraded":"2024-09-22T03:02:16.733274+0000","last_fullsized":"2024-09-22T03:02:16.733274+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2120,"log_dups_size":0,"ondisk_log_size":2120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117768192,"num_objects":1797,"num_object_clones":0,"num_object_copies":5391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2121,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":85,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.235603+0000","last_change":"2024-09-22T03:02:15.236442+0000","last_active":"2024-09-22T03:02:15.235603+0000","last_peered":"2024-09-22T03:02:15.235603+0000","last_clean":"2024-09-22T03:01:41.132979+0000","last_became_active":"2024-09-22T03:02:15.230476+0000","last_became_peered":"2024-09-22T03:02:15.230476+0000","last_unstale":"2024-09-22T03:02:15.235603+0000","last_undegraded":"2024-09-22T03:02:15.235603+0000","last_fullsized":"2024-09-22T03:02:15.235603+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"32'2008","reported_seq":2080,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-22T03:01:46.603734+0000","last_change":"2024-09-22T03:02:13.121391+0000","last_active":"2024-09-22T03:02:13.121391+0000","last_peered":"2024-09-22T03:01:46.603734+0000","last_clean":"2024-09-22T03:01:46.603734+0000","last_became_active":"2024-09-22T03:01:39.217644+0000","last_became_peered":"2024-09-22T03:01:39.217644+0000","last_unstale":"2024-09-22T03:01:46.603734+0000","last_undegraded":"2024-09-22T03:01:46.603734+0000","last_fullsized":"2024-09-22T03:01:46.603734+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2008,"log_dups_size":0,"ondisk_log_size":2008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115474432,"num_objects":1762,"num_object_clones":0,"num_object_copies":5286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2008,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":65,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-22T03:01:41.161257+0000","last_change":"2024-09-22T03:02:13.110401+0000","last_active":"2024-09-22T03:02:13.110401+0000","last_peered":"2024-09-22T03:01:41.161257+0000","last_clean":"2024-09-22T03:01:41.161257+0000","last_became_active":"2024-09-22T03:01:39.187168+0000","last_became_peered":"2024-09-22T03:01:39.187168+0000","last_unstale":"2024-09-22T03:01:41.161257+0000","last_undegraded":"2024-09-22T03:01:41.161257+0000","last_fullsized":"2024-09-22T03:01:41.161257+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":208,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.187171+0000","last_change":"2024-09-22T03:02:15.188071+0000","last_active":"2024-09-22T03:02:15.187171+0000","last_peered":"2024-09-22T03:02:15.187171+0000","last_clean":"2024-09-22T03:01:41.115297+0000","last_became_active":"2024-09-22T03:02:15.178839+0000","last_became_peered":"2024-09-22T03:02:15.178839+0000","last_unstale":"2024-09-22T03:02:15.187171+0000","last_undegraded":"2024-09-22T03:02:15.187171+0000","last_fullsized":"2024-09-22T03:02:15.187171+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"32'2172","reported_seq":2238,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-22T03:01:46.595971+0000","last_change":"2024-09-22T03:02:13.099974+0000","last_active":"2024-09-22T03:02:13.099974+0000","last_peered":"2024-09-22T03:01:46.595971+0000","last_clean":"2024-09-22T03:01:46.595971+0000","last_became_active":"2024-09-22T03:01:39.188561+0000","last_became_peered":"2024-09-22T03:01:39.188561+0000","last_unstale":"2024-09-22T03:01:46.595971+0000","last_undegraded":"2024-09-22T03:01:46.595971+0000","last_fullsized":"2024-09-22T03:01:46.595971+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2172,"log_dups_size":0,"ondisk_log_size":2172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123600896,"num_objects":1886,"num_object_clones":0,"num_object_copies":5658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2172,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"35'2120","reported_seq":2190,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:16.789810+0000","last_change":"2024-09-22T03:02:15.233604+0000","last_active":"2024-09-22T03:02:16.789810+0000","last_peered":"2024-09-22T03:02:16.789810+0000","last_clean":"2024-09-22T03:02:16.789810+0000","last_became_active":"2024-09-22T03:02:15.227759+0000","last_became_peered":"2024-09-22T03:02:15.227759+0000","last_unstale":"2024-09-22T03:02:16.789810+0000","last_undegraded":"2024-09-22T03:02:16.789810+0000","last_fullsized":"2024-09-22T03:02:16.789810+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2120,"log_dups_size":0,"ondisk_log_size":2120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118161408,"num_objects":1803,"num_object_clones":0,"num_object_copies":5409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2121,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":44,"reported_epoch":32,"state":"stale+active+clean","last_fresh":"2024-09-22T03:01:41.116616+0000","last_change":"2024-09-22T03:01:37.133459+0000","last_active":"2024-09-22T03:01:41.116616+0000","last_peered":"2024-09-22T03:01:41.116616+0000","last_clean":"2024-09-22T03:01:41.116616+0000","last_became_active":"2024-09-22T03:01:03.184548+0000","last_became_peered":"2024-09-22T03:01:03.184548+0000","last_unstale":"2024-09-22T03:02:13.091042+0000","last_undegraded":"2024-09-22T03:01:41.116616+0000","last_fullsized":"2024-09-22T03:01:41.116616+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"32'2126","reported_seq":2190,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-22T03:01:46.589155+0000","last_change":"2024-09-22T03:02:13.111555+0000","last_active":"2024-09-22T03:02:13.111555+0000","last_peered":"2024-09-22T03:01:46.589155+0000","last_clean":"2024-09-22T03:01:46.589155+0000","last_became_active":"2024-09-22T03:01:39.191156+0000","last_became_peered":"2024-09-22T03:01:39.191156+0000","last_unstale":"2024-09-22T03:01:46.589155+0000","last_undegraded":"2024-09-22T03:01:46.589155+0000","last_fullsized":"2024-09-22T03:01:46.589155+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2126,"log_dups_size":0,"ondisk_log_size":2126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121241600,"num_objects":1850,"num_object_clones":0,"num_object_copies":5550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2126,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.106352+0000","last_change":"2024-09-22T03:01:40.122900+0000","last_active":"2024-09-22T03:02:15.106352+0000","last_peered":"2024-09-22T03:02:15.106352+0000","last_clean":"2024-09-22T03:02:15.106352+0000","last_became_active":"2024-09-22T03:01:39.184707+0000","last_became_peered":"2024-09-22T03:01:39.184707+0000","last_unstale":"2024-09-22T03:02:15.106352+0000","last_undegraded":"2024-09-22T03:02:15.106352+0000","last_fullsized":"2024-09-22T03:02:15.106352+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"35'2057","reported_seq":2125,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:16.872297+0000","last_change":"2024-09-22T03:01:39.195499+0000","last_active":"2024-09-22T03:02:16.872297+0000","last_peered":"2024-09-22T03:02:16.872297+0000","last_clean":"2024-09-22T03:02:16.872297+0000","last_became_active":"2024-09-22T03:01:39.181260+0000","last_became_peered":"2024-09-22T03:01:39.181260+0000","last_unstale":"2024-09-22T03:02:16.872297+0000","last_undegraded":"2024-09-22T03:02:16.872297+0000","last_fullsized":"2024-09-22T03:02:16.872297+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":1728,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2058,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-22T03:01:39.120616+0000","last_change":"2024-09-22T03:01:37.125439+0000","last_active":"2024-09-22T03:01:37.125439+0000","last_peered":"2024-09-22T03:01:03.203977+0000","last_clean":"2024-09-22T03:01:03.203977+0000","last_became_active":"2024-09-22T03:01:03.197400+0000","last_became_peered":"2024-09-22T03:01:03.197400+0000","last_unstale":"2024-09-22T03:01:39.120616+0000","last_undegraded":"2024-09-22T03:01:39.120616+0000","last_fullsized":"2024-09-22T03:01:39.120616+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"32'1947","reported_seq":2007,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-22T03:01:46.549960+0000","last_change":"2024-09-22T03:02:13.111786+0000","last_active":"2024-09-22T03:02:13.111786+0000","last_peered":"2024-09-22T03:01:46.549960+0000","last_clean":"2024-09-22T03:01:46.549960+0000","last_became_active":"2024-09-22T03:01:39.191673+0000","last_became_peered":"2024-09-22T03:01:39.191673+0000","last_unstale":"2024-09-22T03:01:46.549960+0000","last_undegraded":"2024-09-22T03:01:46.549960+0000","last_fullsized":"2024-09-22T03:01:46.549960+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":1947,"log_dups_size":0,"ondisk_log_size":1947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111214592,"num_objects":1697,"num_object_clones":0,"num_object_copies":5091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1947,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":32,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-22T03:01:39.118355+0000","last_change":"2024-09-22T03:01:37.127566+0000","last_active":"2024-09-22T03:01:37.127566+0000","last_peered":"2024-09-22T03:01:03.149037+0000","last_clean":"2024-09-22T03:01:03.149037+0000","last_became_active":"2024-09-22T03:00:26.202803+0000","last_became_peered":"2024-09-22T03:00:26.202803+0000","last_unstale":"2024-09-22T03:01:39.118355+0000","last_undegraded":"2024-09-22T03:01:39.118355+0000","last_fullsized":"2024-09-22T03:01:39.118355+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1856831512,"num_objects":28334,"num_object_clones":0,"num_object_copies":85002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":33078,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348925952,"data_stored":5582618696,"data_compressed":265092602,"data_compressed_allocated":348913664,"data_compressed_original":5582618624,"omap_allocated":0,"internal_metadata":0},"log_size":33069,"ondisk_log_size":33069,"up":42,"acting":42,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":21,"acting":21,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":29,"seq":124554051619,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":136568,"kb_used_data":100648,"kb_used_omap":0,"kb_used_meta":35904,"kb_avail":94235272,"statfs":{"total":96636764160,"available":96496918528,"internally_reserved":0,"allocated":103063552,"data_stored":1644592486,"data_compressed":78089415,"data_compressed_allocated":102780928,"data_compressed_original":1644494848,"omap_allocated":0,"internal_metadata":36765696},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":26,"seq":111669149770,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":111924,"kb_used_data":77412,"kb_used_omap":0,"kb_used_meta":34496,"kb_avail":94259916,"statfs":{"total":96636764160,"available":96522153984,"internally_reserved":0,"allocated":79269888,"data_stored":1263311781,"data_compressed":59985750,"data_compressed_allocated":78954496,"data_compressed_original":1263190016,"omap_allocated":0,"internal_metadata":35323904},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673174,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":127264,"kb_used_data":78224,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94244576,"statfs":{"total":96636764160,"available":96506445824,"internally_reserved":0,"allocated":80101376,"data_stored":1276615594,"data_compressed":60617485,"data_compressed_allocated":79785984,"data_compressed_original":1276493824,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126220,"kb_used_data":87800,"kb_used_omap":0,"kb_used_meta":38400,"kb_avail":94245620,"statfs":{"total":96636764160,"available":96507514880,"internally_reserved":0,"allocated":89907200,"data_stored":1433825582,"data_compressed":68083173,"data_compressed_allocated":89612288,"data_compressed_original":1433714688,"omap_allocated":0,"internal_metadata":39321600},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":78999552,"data_stored":1263927320,"data_compressed":60018030,"data_compressed_allocated":78995456,"data_compressed_original":1263927296,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88866816,"data_stored":1421803544,"data_compressed":67514838,"data_compressed_allocated":88862720,"data_compressed_original":1421803520,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":78274560,"data_stored":1252392960,"data_compressed":59470319,"data_compressed_allocated":78274560,"data_compressed_original":1252392960,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102785024,"data_stored":1644494872,"data_compressed":78089415,"data_compressed_allocated":102780928,"data_compressed_original":1644494848,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:02:19.984 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:02:20.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:20.132+0000 7fd638d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:20.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:20.139+0000 7fd638d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:22.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:02:22.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:22.515+0000 7fc6e9397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:22.522+0000 7fc6e9397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:23.543 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-09-22T03:02:23.543 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:02:23.543 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:02:23.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:23.691+0000 7fee6ac2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:23.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:23.701+0000 7fee6ac2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:23.995 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:02:23.995 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:02:24.014 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-22T03:02:22.327529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1797128779,"num_objects":27418,"num_object_clones":0,"num_object_copies":82254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":34137,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34082,"ondisk_log_size":34082,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":504100,"kb_used_data":332644,"kb_used_omap":0,"kb_used_meta":171392,"kb_avail":376983260,"statfs":{"total":386547056640,"available":386030858240,"internally_reserved":0,"allocated":340627456,"data_stored":5430275241,"data_compressed":257844383,"data_compressed_allocated":339378176,"data_compressed_original":5429805056,"omap_allocated":0,"internal_metadata":175505408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-78315520,"num_objects":-1195,"num_object_clones":0,"num_object_copies":-3585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1195,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.235209"},"pg_stats":[{"pgid":"3.f","version":"35'2109","reported_seq":2192,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:17.121501+0000","last_change":"2024-09-22T03:02:15.329405+0000","last_active":"2024-09-22T03:02:17.121501+0000","last_peered":"2024-09-22T03:02:17.121501+0000","last_clean":"2024-09-22T03:02:17.121501+0000","last_became_active":"2024-09-22T03:02:15.205488+0000","last_became_peered":"2024-09-22T03:02:15.205488+0000","last_unstale":"2024-09-22T03:02:17.121501+0000","last_undegraded":"2024-09-22T03:02:17.121501+0000","last_fullsized":"2024-09-22T03:02:17.121501+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2109,"log_dups_size":0,"ondisk_log_size":2109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116129792,"num_objects":1772,"num_object_clones":0,"num_object_copies":5316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2110,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"35'2176","reported_seq":2261,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:21.840990+0000","last_change":"2024-09-22T03:02:15.425881+0000","last_active":"2024-09-22T03:02:21.840990+0000","last_peered":"2024-09-22T03:02:21.840990+0000","last_clean":"2024-09-22T03:02:21.840990+0000","last_became_active":"2024-09-22T03:02:15.201857+0000","last_became_peered":"2024-09-22T03:02:15.201857+0000","last_unstale":"2024-09-22T03:02:21.840990+0000","last_undegraded":"2024-09-22T03:02:21.840990+0000","last_fullsized":"2024-09-22T03:02:21.840990+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2176,"log_dups_size":0,"ondisk_log_size":2176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108068864,"num_objects":1649,"num_object_clones":0,"num_object_copies":4947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2177,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"35'2102","reported_seq":2176,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:18.789906+0000","last_change":"2024-09-22T03:02:15.303999+0000","last_active":"2024-09-22T03:02:18.789906+0000","last_peered":"2024-09-22T03:02:18.789906+0000","last_clean":"2024-09-22T03:02:18.789906+0000","last_became_active":"2024-09-22T03:02:15.270675+0000","last_became_peered":"2024-09-22T03:02:15.270675+0000","last_unstale":"2024-09-22T03:02:18.789906+0000","last_undegraded":"2024-09-22T03:02:18.789906+0000","last_fullsized":"2024-09-22T03:02:18.789906+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114819096,"num_objects":1753,"num_object_clones":0,"num_object_copies":5259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2103,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"35'2012","reported_seq":2091,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:17.298501+0000","last_change":"2024-09-22T03:01:39.254261+0000","last_active":"2024-09-22T03:02:17.298501+0000","last_peered":"2024-09-22T03:02:17.298501+0000","last_clean":"2024-09-22T03:02:17.298501+0000","last_became_active":"2024-09-22T03:01:39.165796+0000","last_became_peered":"2024-09-22T03:01:39.165796+0000","last_unstale":"2024-09-22T03:02:17.298501+0000","last_undegraded":"2024-09-22T03:02:17.298501+0000","last_fullsized":"2024-09-22T03:02:17.298501+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2012,"log_dups_size":0,"ondisk_log_size":2012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110166016,"num_objects":1681,"num_object_clones":0,"num_object_copies":5043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2013,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"35'2106","reported_seq":2185,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:18.095747+0000","last_change":"2024-09-22T03:01:39.285980+0000","last_active":"2024-09-22T03:02:18.095747+0000","last_peered":"2024-09-22T03:02:18.095747+0000","last_clean":"2024-09-22T03:02:18.095747+0000","last_became_active":"2024-09-22T03:01:39.191952+0000","last_became_peered":"2024-09-22T03:01:39.191952+0000","last_unstale":"2024-09-22T03:02:18.095747+0000","last_undegraded":"2024-09-22T03:02:18.095747+0000","last_fullsized":"2024-09-22T03:02:18.095747+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111607808,"num_objects":1703,"num_object_clones":0,"num_object_copies":5109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2107,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"35'2052","reported_seq":2129,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:17.253536+0000","last_change":"2024-09-22T03:02:15.196410+0000","last_active":"2024-09-22T03:02:17.253536+0000","last_peered":"2024-09-22T03:02:17.253536+0000","last_clean":"2024-09-22T03:02:17.253536+0000","last_became_active":"2024-09-22T03:02:15.193360+0000","last_became_peered":"2024-09-22T03:02:15.193360+0000","last_unstale":"2024-09-22T03:02:17.253536+0000","last_undegraded":"2024-09-22T03:02:17.253536+0000","last_fullsized":"2024-09-22T03:02:17.253536+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113836032,"num_objects":1737,"num_object_clones":0,"num_object_copies":5211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2053,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"35'2040","reported_seq":2115,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:17.234556+0000","last_change":"2024-09-22T03:02:15.183663+0000","last_active":"2024-09-22T03:02:17.234556+0000","last_peered":"2024-09-22T03:02:17.234556+0000","last_clean":"2024-09-22T03:02:17.234556+0000","last_became_active":"2024-09-22T03:02:15.175035+0000","last_became_peered":"2024-09-22T03:02:15.175035+0000","last_unstale":"2024-09-22T03:02:17.234556+0000","last_undegraded":"2024-09-22T03:02:17.234556+0000","last_fullsized":"2024-09-22T03:02:17.234556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110690304,"num_objects":1689,"num_object_clones":0,"num_object_copies":5067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2041,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"35'2162","reported_seq":2236,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:18.650174+0000","last_change":"2024-09-22T03:02:15.291580+0000","last_active":"2024-09-22T03:02:18.650174+0000","last_peered":"2024-09-22T03:02:18.650174+0000","last_clean":"2024-09-22T03:02:18.650174+0000","last_became_active":"2024-09-22T03:02:15.268207+0000","last_became_peered":"2024-09-22T03:02:15.268207+0000","last_unstale":"2024-09-22T03:02:18.650174+0000","last_undegraded":"2024-09-22T03:02:18.650174+0000","last_fullsized":"2024-09-22T03:02:18.650174+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2162,"log_dups_size":0,"ondisk_log_size":2162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112656384,"num_objects":1719,"num_object_clones":0,"num_object_copies":5157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":81,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.112380+0000","last_change":"2024-09-22T03:01:40.146115+0000","last_active":"2024-09-22T03:02:15.112380+0000","last_peered":"2024-09-22T03:02:15.112380+0000","last_clean":"2024-09-22T03:02:15.112380+0000","last_became_active":"2024-09-22T03:01:39.198270+0000","last_became_peered":"2024-09-22T03:01:39.198270+0000","last_unstale":"2024-09-22T03:02:15.112380+0000","last_undegraded":"2024-09-22T03:02:15.112380+0000","last_fullsized":"2024-09-22T03:02:15.112380+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"35'2144","reported_seq":2218,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:18.757953+0000","last_change":"2024-09-22T03:02:15.397801+0000","last_active":"2024-09-22T03:02:18.757953+0000","last_peered":"2024-09-22T03:02:18.757953+0000","last_clean":"2024-09-22T03:02:18.757953+0000","last_became_active":"2024-09-22T03:02:15.241874+0000","last_became_peered":"2024-09-22T03:02:15.241874+0000","last_unstale":"2024-09-22T03:02:18.757953+0000","last_undegraded":"2024-09-22T03:02:18.757953+0000","last_fullsized":"2024-09-22T03:02:18.757953+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2144,"log_dups_size":0,"ondisk_log_size":2144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112787456,"num_objects":1721,"num_object_clones":0,"num_object_copies":5163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":83,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.164068+0000","last_change":"2024-09-22T03:02:15.164748+0000","last_active":"2024-09-22T03:02:15.164068+0000","last_peered":"2024-09-22T03:02:15.164068+0000","last_clean":"2024-09-22T03:01:41.112371+0000","last_became_active":"2024-09-22T03:02:15.159837+0000","last_became_peered":"2024-09-22T03:02:15.159837+0000","last_unstale":"2024-09-22T03:02:15.164068+0000","last_undegraded":"2024-09-22T03:02:15.164068+0000","last_fullsized":"2024-09-22T03:02:15.164068+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"35'2217","reported_seq":2290,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:21.794377+0000","last_change":"2024-09-22T03:01:39.280325+0000","last_active":"2024-09-22T03:02:21.794377+0000","last_peered":"2024-09-22T03:02:21.794377+0000","last_clean":"2024-09-22T03:02:21.794377+0000","last_became_active":"2024-09-22T03:01:39.204351+0000","last_became_peered":"2024-09-22T03:01:39.204351+0000","last_unstale":"2024-09-22T03:02:21.794377+0000","last_undegraded":"2024-09-22T03:02:21.794377+0000","last_fullsized":"2024-09-22T03:02:21.794377+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2217,"log_dups_size":0,"ondisk_log_size":2217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111411200,"num_objects":1700,"num_object_clones":0,"num_object_copies":5100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2218,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":85,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.235603+0000","last_change":"2024-09-22T03:02:15.236442+0000","last_active":"2024-09-22T03:02:15.235603+0000","last_peered":"2024-09-22T03:02:15.235603+0000","last_clean":"2024-09-22T03:01:41.132979+0000","last_became_active":"2024-09-22T03:02:15.230476+0000","last_became_peered":"2024-09-22T03:02:15.230476+0000","last_unstale":"2024-09-22T03:02:15.235603+0000","last_undegraded":"2024-09-22T03:02:15.235603+0000","last_fullsized":"2024-09-22T03:02:15.235603+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"35'2064","reported_seq":2155,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:18.184302+0000","last_change":"2024-09-22T03:02:15.432234+0000","last_active":"2024-09-22T03:02:18.184302+0000","last_peered":"2024-09-22T03:02:18.184302+0000","last_clean":"2024-09-22T03:02:18.184302+0000","last_became_active":"2024-09-22T03:02:15.236858+0000","last_became_peered":"2024-09-22T03:02:15.236858+0000","last_unstale":"2024-09-22T03:02:18.184302+0000","last_undegraded":"2024-09-22T03:02:18.184302+0000","last_fullsized":"2024-09-22T03:02:18.184302+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2064,"log_dups_size":0,"ondisk_log_size":2064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111738880,"num_objects":1705,"num_object_clones":0,"num_object_copies":5115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2065,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":74,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.216104+0000","last_change":"2024-09-22T03:02:15.217039+0000","last_active":"2024-09-22T03:02:15.216104+0000","last_peered":"2024-09-22T03:02:15.216104+0000","last_clean":"2024-09-22T03:01:41.161257+0000","last_became_active":"2024-09-22T03:02:15.208759+0000","last_became_peered":"2024-09-22T03:02:15.208759+0000","last_unstale":"2024-09-22T03:02:15.216104+0000","last_undegraded":"2024-09-22T03:02:15.216104+0000","last_fullsized":"2024-09-22T03:02:15.216104+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":208,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.187171+0000","last_change":"2024-09-22T03:02:15.188071+0000","last_active":"2024-09-22T03:02:15.187171+0000","last_peered":"2024-09-22T03:02:15.187171+0000","last_clean":"2024-09-22T03:01:41.115297+0000","last_became_active":"2024-09-22T03:02:15.178839+0000","last_became_peered":"2024-09-22T03:02:15.178839+0000","last_unstale":"2024-09-22T03:02:15.187171+0000","last_undegraded":"2024-09-22T03:02:15.187171+0000","last_fullsized":"2024-09-22T03:02:15.187171+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"35'2241","reported_seq":2323,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:18.299237+0000","last_change":"2024-09-22T03:02:15.380756+0000","last_active":"2024-09-22T03:02:18.299237+0000","last_peered":"2024-09-22T03:02:18.299237+0000","last_clean":"2024-09-22T03:02:18.299237+0000","last_became_active":"2024-09-22T03:02:15.214276+0000","last_became_peered":"2024-09-22T03:02:15.214276+0000","last_unstale":"2024-09-22T03:02:18.299237+0000","last_undegraded":"2024-09-22T03:02:18.299237+0000","last_fullsized":"2024-09-22T03:02:18.299237+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2241,"log_dups_size":0,"ondisk_log_size":2241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119013376,"num_objects":1816,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2242,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"35'2210","reported_seq":2280,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:21.806987+0000","last_change":"2024-09-22T03:02:15.233604+0000","last_active":"2024-09-22T03:02:21.806987+0000","last_peered":"2024-09-22T03:02:21.806987+0000","last_clean":"2024-09-22T03:02:21.806987+0000","last_became_active":"2024-09-22T03:02:15.227759+0000","last_became_peered":"2024-09-22T03:02:15.227759+0000","last_unstale":"2024-09-22T03:02:21.806987+0000","last_undegraded":"2024-09-22T03:02:21.806987+0000","last_fullsized":"2024-09-22T03:02:21.806987+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112263168,"num_objects":1713,"num_object_clones":0,"num_object_copies":5139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2211,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.210993+0000","last_change":"2024-09-22T03:02:15.211495+0000","last_active":"2024-09-22T03:02:15.210993+0000","last_peered":"2024-09-22T03:02:15.210993+0000","last_clean":"2024-09-22T03:01:03.188621+0000","last_became_active":"2024-09-22T03:02:15.202530+0000","last_became_peered":"2024-09-22T03:02:15.202530+0000","last_unstale":"2024-09-22T03:02:15.210993+0000","last_undegraded":"2024-09-22T03:02:15.210993+0000","last_fullsized":"2024-09-22T03:02:15.210993+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"35'2183","reported_seq":2263,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:18.144565+0000","last_change":"2024-09-22T03:02:15.356780+0000","last_active":"2024-09-22T03:02:18.144565+0000","last_peered":"2024-09-22T03:02:18.144565+0000","last_clean":"2024-09-22T03:02:18.144565+0000","last_became_active":"2024-09-22T03:02:15.207746+0000","last_became_peered":"2024-09-22T03:02:15.207746+0000","last_unstale":"2024-09-22T03:02:18.144565+0000","last_undegraded":"2024-09-22T03:02:18.144565+0000","last_fullsized":"2024-09-22T03:02:18.144565+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":1792,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":44,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.106352+0000","last_change":"2024-09-22T03:01:40.122900+0000","last_active":"2024-09-22T03:02:15.106352+0000","last_peered":"2024-09-22T03:02:15.106352+0000","last_clean":"2024-09-22T03:02:15.106352+0000","last_became_active":"2024-09-22T03:01:39.184707+0000","last_became_peered":"2024-09-22T03:01:39.184707+0000","last_unstale":"2024-09-22T03:02:15.106352+0000","last_undegraded":"2024-09-22T03:02:15.106352+0000","last_fullsized":"2024-09-22T03:02:15.106352+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"35'2158","reported_seq":2226,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:21.707413+0000","last_change":"2024-09-22T03:01:39.195499+0000","last_active":"2024-09-22T03:02:21.707413+0000","last_peered":"2024-09-22T03:02:21.707413+0000","last_clean":"2024-09-22T03:02:21.707413+0000","last_became_active":"2024-09-22T03:01:39.181260+0000","last_became_peered":"2024-09-22T03:01:39.181260+0000","last_unstale":"2024-09-22T03:02:21.707413+0000","last_undegraded":"2024-09-22T03:02:21.707413+0000","last_fullsized":"2024-09-22T03:02:21.707413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2158,"log_dups_size":0,"ondisk_log_size":2158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106627072,"num_objects":1627,"num_object_clones":0,"num_object_copies":4881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2159,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":47,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.308694+0000","last_change":"2024-09-22T03:02:15.309567+0000","last_active":"2024-09-22T03:02:15.308694+0000","last_peered":"2024-09-22T03:02:15.308694+0000","last_clean":"2024-09-22T03:01:39.211883+0000","last_became_active":"2024-09-22T03:02:15.289741+0000","last_became_peered":"2024-09-22T03:02:15.289741+0000","last_unstale":"2024-09-22T03:02:15.308694+0000","last_undegraded":"2024-09-22T03:02:15.308694+0000","last_fullsized":"2024-09-22T03:02:15.308694+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"35'2006","reported_seq":2075,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:18.748029+0000","last_change":"2024-09-22T03:02:15.245512+0000","last_active":"2024-09-22T03:02:18.748029+0000","last_peered":"2024-09-22T03:02:18.748029+0000","last_clean":"2024-09-22T03:02:18.748029+0000","last_became_active":"2024-09-22T03:02:15.236668+0000","last_became_peered":"2024-09-22T03:02:15.236668+0000","last_unstale":"2024-09-22T03:02:18.748029+0000","last_undegraded":"2024-09-22T03:02:18.748029+0000","last_fullsized":"2024-09-22T03:02:18.748029+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2006,"log_dups_size":0,"ondisk_log_size":2006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107282432,"num_objects":1637,"num_object_clones":0,"num_object_copies":4911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2007,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":47,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-22T03:02:15.256276+0000","last_change":"2024-09-22T03:02:15.256754+0000","last_active":"2024-09-22T03:02:15.256276+0000","last_peered":"2024-09-22T03:02:15.256276+0000","last_clean":"2024-09-22T03:01:39.214571+0000","last_became_active":"2024-09-22T03:02:15.252373+0000","last_became_peered":"2024-09-22T03:02:15.252373+0000","last_unstale":"2024-09-22T03:02:15.256276+0000","last_undegraded":"2024-09-22T03:02:15.256276+0000","last_fullsized":"2024-09-22T03:02:15.256276+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1796538392,"num_objects":27414,"num_object_clones":0,"num_object_copies":82242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":33998,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":337117184,"data_stored":5393678408,"data_compressed":256120706,"data_compressed_allocated":337104896,"data_compressed_original":5393678336,"omap_allocated":0,"internal_metadata":0},"log_size":33982,"ondisk_log_size":33982,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":29,"seq":124554051624,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":137580,"kb_used_data":97500,"kb_used_omap":0,"kb_used_meta":40064,"kb_avail":94234260,"statfs":{"total":96636764160,"available":96495882240,"internally_reserved":0,"allocated":99840000,"data_stored":1592760733,"data_compressed":75627823,"data_compressed_allocated":99540992,"data_compressed_original":1592655872,"omap_allocated":0,"internal_metadata":41025536},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":26,"seq":111669149775,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":111800,"kb_used_data":73128,"kb_used_omap":0,"kb_used_meta":38656,"kb_avail":94260040,"statfs":{"total":96636764160,"available":96522280960,"internally_reserved":0,"allocated":74883072,"data_stored":1193122725,"data_compressed":56652798,"data_compressed_allocated":74567680,"data_compressed_original":1193000960,"omap_allocated":0,"internal_metadata":39583744},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673174,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":127264,"kb_used_data":78224,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94244576,"statfs":{"total":96636764160,"available":96506445824,"internally_reserved":0,"allocated":80101376,"data_stored":1276615594,"data_compressed":60617485,"data_compressed_allocated":79785984,"data_compressed_original":1276493824,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888068,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":127456,"kb_used_data":83792,"kb_used_omap":0,"kb_used_meta":43648,"kb_avail":94244384,"statfs":{"total":96636764160,"available":96506249216,"internally_reserved":0,"allocated":85803008,"data_stored":1367776189,"data_compressed":64946277,"data_compressed_allocated":85483520,"data_compressed_original":1367654400,"omap_allocated":0,"internal_metadata":44695552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":78999552,"data_stored":1263927320,"data_compressed":60018030,"data_compressed_allocated":78995456,"data_compressed_original":1263927296,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":84791296,"data_stored":1356595224,"data_compressed":64418398,"data_compressed_allocated":84787200,"data_compressed_original":1356595200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73945088,"data_stored":1183121408,"data_compressed":56180935,"data_compressed_allocated":73945088,"data_compressed_original":1183121408,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99381248,"data_stored":1590034456,"data_compressed":75503343,"data_compressed_allocated":99377152,"data_compressed_original":1590034432,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:02:24.878 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:02:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:25.010+0000 7f4111597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:25.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:25.016+0000 7f4111597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:27.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:02:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:27.469+0000 7fb644597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:27.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:27.476+0000 7fb644597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:29.016 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-09-22T03:02:29.017 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:02:29.017 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:02:29.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:02:29.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:29.167+0000 7f3d912ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:29.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:29.178+0000 7f3d912ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:29.255 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:02:29.255 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:02:29.255 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:02:29.255 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:02:29.256 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:02:29.256 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:02:29.256 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:02:29.256 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:02:29.256 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:02:29.256 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:02:29.256 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:02:29.256 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:02:29.259 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:02:29.259 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:02:29.259 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:02:29.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:29.403+0000 7fd42e397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:29.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:29.416+0000 7fd42e397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:02:29.544 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:02:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:02:29.548 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:02:29.548 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:02:29.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:29.687+0000 7f5b13157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:29.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:29.699+0000 7f5b13157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:29.769 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:02:29.769 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:02:29.769 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:02:29.769 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:02:29.769 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:02:29.770 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:02:29.770 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:02:29.770 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:02:29.770 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:02:29.770 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:02:29.770 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:02:29.770 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:02:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:02:29.773 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:02:29.774 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:02:29.838 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:02:29.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:29.900+0000 7f8e4c317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:29.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:29.916+0000 7f8e4c317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:02:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:02:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:02:29.960 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:02:29.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:29.988+0000 7f74f6b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:29.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:29.994+0000 7f74f6b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:32.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:02:32.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:32.564+0000 7f3dcfad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:32.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:32.571+0000 7f3dcfad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:34.943 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:02:34.961 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-09-22T03:02:34.961 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:02:34.961 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 2, 3, 1] 2024-09-22T03:02:34.961 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:02:34.961 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:02:35.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:35.044+0000 7f62dfe2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:35.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:35.048+0000 7f62dfe2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:37.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-22T03:02:41.062 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:02:43.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-22T03:02:46.063 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 2, 1] 2024-09-22T03:02:46.063 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:02:46.063 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 2, 1] 2024-09-22T03:02:46.063 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:02:46.063 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:02:48.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:02:52.164 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:02:54.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-22T03:02:57.165 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 2] live_osds: [0, 1] 2024-09-22T03:02:57.165 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:02:57.165 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:02:57.166 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:02:57.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:57.279+0000 7f4987357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:57.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:57.286+0000 7f4987357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:02:57.300 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:02:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:02:57.303 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:02:57.303 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:02:57.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:57.404+0000 7ffa45997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:57.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:57.412+0000 7ffa45997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:02:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:02:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:02:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:02:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:02:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:02:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:02:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:02:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:02:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:02:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:02:57.428 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:02:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:02:57.430 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:02:57.430 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:02:57.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:57.529+0000 7f0d05cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:02:57.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:02:57.537+0000 7f0d05cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:01.401 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:01.407 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:03.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:03:03.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:03.512+0000 7efd77f57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:03.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:03.516+0000 7efd77f57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:13.417 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:13.418 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:13.418 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:13.424 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:13.424 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:13.424 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:03:13.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:03:13.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:13.544+0000 7f72504b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:13.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:13.551+0000 7f72504b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:13.556 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:13.559 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:03:15.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:03:15.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:15.530+0000 7fe778557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:15.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:15.535+0000 7fe778557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:15.566 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:15.566 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:15.570 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:17.571 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:03:17.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:17.686+0000 7f7818e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:17.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:17.690+0000 7f7818e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:17.723 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:17.723 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:17.727 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:18.560 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 2] live_osds: [0, 1] 2024-09-22T03:03:18.561 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:03:18.561 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-22T03:03:18.561 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:03:18.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:03:18.563 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:03:18.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:03:18.673 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:03:18.676 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:03:18.677 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:03:19.728 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:03:19.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:19.840+0000 7f9c7fd57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:19.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:19.845+0000 7f9c7fd57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:19.876 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:19.876 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:19.880 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:19.880 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:21.881 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:03:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:21.992+0000 7fc9a9cb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:21.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:21.997+0000 7fc9a9cb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:22.029 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:22.029 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:22.034 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:23.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:03:23.773 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:03:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:03:23.776 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:03:24.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:03:24.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:24.151+0000 7f16d9d0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:24.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:24.156+0000 7f16d9d0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:24.187 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:24.187 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:24.191 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:26.192 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:03:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:26.304+0000 7fd646d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:26.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:26.308+0000 7fd646d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:26.340 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:26.341 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:26.344 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:28.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:03:28.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:28.458+0000 7fbf4924f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:28.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:28.463+0000 7fbf4924f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:28.495 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:28.495 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:28.499 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:28.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:03:28.871 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:03:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:03:28.874 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:03:30.500 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:03:30.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:30.613+0000 7feed3e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:30.617+0000 7feed3e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:30.649 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:30.649 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:30.653 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:30.653 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:32.654 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:03:32.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:32.766+0000 7f81d3797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:32.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:32.770+0000 7f81d3797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:32.802 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:32.802 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:03:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:32.807 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:33.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:03:33.973 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:03:33.976 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:03:33.976 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:03:34.808 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:03:34.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:34.908+0000 7f81a654e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:34.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:34.913+0000 7f81a654e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:34.944 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:34.962 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:34.963 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:36.963 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:03:37.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:37.121+0000 7f8c39797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:37.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:37.129+0000 7f8c39797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:37.275 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:37.339 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:38.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:03:39.153 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-22T03:03:39.154 INFO:teuthology.orchestra.run.smithi190.stdout: "ops_in_flight": [ 2024-09-22T03:03:39.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.154 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.154 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778040, 2024-09-22T03:03:39.154 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.154 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.154 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.075350+0000" 2024-09-22T03:03:39.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.154 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.154 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.075355+0000" 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.075414+0000" 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.075466+0000" 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.156 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.077474+0000" 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.077843+0000" 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.158 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.078144+0000" 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.160 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.078320+0000" 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.162 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.078510+0000" 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.163 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.080898+0000" 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.165 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778044, 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.099501+0000" 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.166 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.099507+0000" 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.099586+0000" 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.167 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.099649+0000" 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.168 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.102960+0000" 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.103347+0000" 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.103614+0000" 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.103745+0000" 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.103934+0000" 2024-09-22T03:03:39.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.175 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.177 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777974, 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.693421+0000" 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.693424+0000" 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.693502+0000" 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.179 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.693541+0000" 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.693872+0000" 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.735537+0000" 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.181 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.694082+0000" 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.694177+0000" 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.183 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.694276+0000" 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.184 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.694413+0000" 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.186 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.695516+0000" 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.735310+0000" 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777978, 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.189 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.733342+0000" 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.733348+0000" 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.733433+0000" 2024-09-22T03:03:39.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.191 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.733494+0000" 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.733982+0000" 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.192 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.778008+0000" 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.193 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.734244+0000" 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.194 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.734363+0000" 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.734451+0000" 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.196 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.197 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.734579+0000" 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.736145+0000" 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.199 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.775883+0000" 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777982, 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.201 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.741246+0000" 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.741250+0000" 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.741328+0000" 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.203 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.741372+0000" 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.742270+0000" 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.204 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.779177+0000" 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.742668+0000" 2024-09-22T03:03:39.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.206 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.742838+0000" 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.207 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.742989+0000" 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.208 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.743125+0000" 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.210 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.744571+0000" 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.211 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.777849+0000" 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777984, 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.768316+0000" 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.768326+0000" 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.768447+0000" 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.214 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.768525+0000" 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.772077+0000" 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.827448+0000" 2024-09-22T03:03:39.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.772502+0000" 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.217 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.775843+0000" 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.218 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.776006+0000" 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.219 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.220 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.776147+0000" 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.221 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.780426+0000" 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.827119+0000" 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.223 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777992, 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.811697+0000" 2024-09-22T03:03:39.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.811703+0000" 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.811785+0000" 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.225 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.811853+0000" 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.812843+0000" 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.850892+0000" 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.813184+0000" 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.228 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.813901+0000" 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.229 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.230 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.814018+0000" 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.814406+0000" 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.233 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.816418+0000" 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.234 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.850433+0000" 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.235 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777993, 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.236 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.812052+0000" 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.812056+0000" 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.812110+0000" 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.812160+0000" 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.812932+0000" 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.239 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.857045+0000" 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.240 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.814181+0000" 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.241 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.814368+0000" 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.242 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.814521+0000" 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.243 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.814934+0000" 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.820717+0000" 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.856109+0000" 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.247 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777998, 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.855714+0000" 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.855720+0000" 2024-09-22T03:03:39.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.855798+0000" 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.249 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.855851+0000" 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.250 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.856483+0000" 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.891951+0000" 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.856698+0000" 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.252 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.857192+0000" 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.857385+0000" 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.254 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.255 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.857569+0000" 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.256 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.858710+0000" 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.891375+0000" 2024-09-22T03:03:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778004, 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.886127+0000" 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.886133+0000" 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.886255+0000" 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.886304+0000" 2024-09-22T03:03:39.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.889998+0000" 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.262 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.917816+0000" 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.891356+0000" 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.264 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.891541+0000" 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.265 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.891641+0000" 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.266 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.267 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.891908+0000" 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.894113+0000" 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.917548+0000" 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.270 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778006, 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.894617+0000" 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.271 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.894623+0000" 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.894698+0000" 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.894745+0000" 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.273 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.898786+0000" 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.940650+0000" 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.899085+0000" 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.899186+0000" 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.276 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.899293+0000" 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.279 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.900189+0000" 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.280 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.902040+0000" 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.281 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.937045+0000" 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.282 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778010, 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.918429+0000" 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.918434+0000" 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.918520+0000" 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.918567+0000" 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.918820+0000" 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.286 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.958019+0000" 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.919124+0000" 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.919262+0000" 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.288 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.919364+0000" 2024-09-22T03:03:39.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.290 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.291 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.919491+0000" 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.920406+0000" 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.293 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.953151+0000" 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778012, 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.295 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.925014+0000" 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.925020+0000" 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.925133+0000" 2024-09-22T03:03:39.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.297 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.297 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.297 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.297 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.297 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.925191+0000" 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.298 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.926380+0000" 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.973261+0000" 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.930110+0000" 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.300 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.930317+0000" 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.301 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.930479+0000" 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.302 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.304 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.930656+0000" 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.305 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.932743+0000" 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.306 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.963564+0000" 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778015, 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.937627+0000" 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.937633+0000" 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.937696+0000" 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.309 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.937753+0000" 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.310 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.940974+0000" 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.975780+0000" 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.942020+0000" 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.942124+0000" 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.313 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.942215+0000" 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.315 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.942817+0000" 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.944238+0000" 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.318 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.973338+0000" 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.319 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778016, 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.944246+0000" 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.944252+0000" 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.320 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.944323+0000" 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.944369+0000" 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.948640+0000" 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.322 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.323 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.988533+0000" 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.948986+0000" 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.949180+0000" 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.950215+0000" 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.326 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.327 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.950376+0000" 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.953366+0000" 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.988265+0000" 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778018, 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.945948+0000" 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.945951+0000" 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.946031+0000" 2024-09-22T03:03:39.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.333 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.946092+0000" 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.949064+0000" 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.000186+0000" 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.950614+0000" 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.335 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.336 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.951656+0000" 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.337 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.951754+0000" 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.338 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.952027+0000" 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.959778+0000" 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.999898+0000" 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.342 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778020, 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.955987+0000" 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.343 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.955992+0000" 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.956066+0000" 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.344 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.956141+0000" 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.345 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.958870+0000" 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.001176+0000" 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.959507+0000" 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.347 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.963550+0000" 2024-09-22T03:03:39.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.349 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.968625+0000" 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.973091+0000" 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.352 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.976219+0000" 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.000589+0000" 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778017, 2024-09-22T03:03:39.355 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.944535+0000" 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.944538+0000" 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.356 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.944606+0000" 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.944649+0000" 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.949023+0000" 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.359 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.004393+0000" 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.950107+0000" 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.950907+0000" 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.951615+0000" 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.362 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.363 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.951924+0000" 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.365 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.958513+0000" 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.004138+0000" 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778026, 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.000657+0000" 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.000663+0000" 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.368 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.000746+0000" 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.000802+0000" 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.370 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.001522+0000" 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.371 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.042319+0000" 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.001722+0000" 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.372 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.002056+0000" 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.002178+0000" 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.374 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.375 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.375 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.375 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.375 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.376 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.002396+0000" 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.377 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.378 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.004940+0000" 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.042041+0000" 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778028, 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.011386+0000" 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.011395+0000" 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.011478+0000" 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.382 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.011537+0000" 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.383 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.012236+0000" 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.065318+0000" 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.012468+0000" 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.385 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.012573+0000" 2024-09-22T03:03:39.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.387 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.012700+0000" 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.388 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.012866+0000" 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.390 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.014411+0000" 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.391 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.064311+0000" 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.392 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778033, 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.031562+0000" 2024-09-22T03:03:39.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.031567+0000" 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.031636+0000" 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.394 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.031704+0000" 2024-09-22T03:03:39.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.033095+0000" 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.396 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.077311+0000" 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.397 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.034063+0000" 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.398 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.034232+0000" 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.399 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.034365+0000" 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.400 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.034519+0000" 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.403 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.036913+0000" 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.404 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.073607+0000" 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.405 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778045, 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.106583+0000" 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.106588+0000" 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.406 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.106658+0000" 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.407 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.106709+0000" 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.408 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.106754+0000" 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.106936+0000" 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.410 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.107076+0000" 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.411 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.107166+0000" 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.413 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.414 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.107309+0000" 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.108146+0000" 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.416 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.417 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777946, 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.536002+0000" 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.418 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.536008+0000" 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.536090+0000" 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.536136+0000" 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.420 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.536189+0000" 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.561596+0000" 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.536380+0000" 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.422 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.536475+0000" 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.536572+0000" 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.424 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.426 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.536716+0000" 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.427 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.537709+0000" 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.428 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.561130+0000" 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.429 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777956, 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.586761+0000" 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.586765+0000" 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.430 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.586841+0000" 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.432 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.432 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.432 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.432 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.432 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.586881+0000" 2024-09-22T03:03:39.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.432 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.432 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.586922+0000" 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.433 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.619807+0000" 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.587096+0000" 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.434 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.587194+0000" 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.435 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.587282+0000" 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.437 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.438 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.587415+0000" 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.439 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.591444+0000" 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.440 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.619407+0000" 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.441 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777967, 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.660609+0000" 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.660617+0000" 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.660711+0000" 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.443 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.660757+0000" 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.444 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.660808+0000" 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.676293+0000" 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.661021+0000" 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.661126+0000" 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.661228+0000" 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.449 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.661372+0000" 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.450 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.662340+0000" 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.451 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.675926+0000" 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777968, 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.666532+0000" 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.453 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.666539+0000" 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.666608+0000" 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.454 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.666676+0000" 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.666904+0000" 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.682126+0000" 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.667202+0000" 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.667415+0000" 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.458 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.667525+0000" 2024-09-22T03:03:39.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.667708+0000" 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.668666+0000" 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.681741+0000" 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777976, 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.718230+0000" 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.718235+0000" 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.718473+0000" 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.718626+0000" 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.721944+0000" 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.741029+0000" 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.722174+0000" 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.722339+0000" 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.722448+0000" 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.722597+0000" 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.724610+0000" 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.740556+0000" 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777977, 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.723141+0000" 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.723147+0000" 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.723292+0000" 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.477 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.723397+0000" 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.723443+0000" 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.771286+0000" 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.723620+0000" 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.723705+0000" 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.723788+0000" 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.723919+0000" 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.729232+0000" 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.770846+0000" 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777985, 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.768959+0000" 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.768963+0000" 2024-09-22T03:03:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.769047+0000" 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.769115+0000" 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.769163+0000" 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.803274+0000" 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.769382+0000" 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.769514+0000" 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.769610+0000" 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.769797+0000" 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.771630+0000" 2024-09-22T03:03:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.799795+0000" 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777989, 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.789772+0000" 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.789779+0000" 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.789859+0000" 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.789906+0000" 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.789966+0000" 2024-09-22T03:03:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.811533+0000" 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.790169+0000" 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.790266+0000" 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.790360+0000" 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.790492+0000" 2024-09-22T03:03:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.791424+0000" 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.811190+0000" 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777994, 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.822766+0000" 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.822772+0000" 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.822862+0000" 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.822914+0000" 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.822972+0000" 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.866198+0000" 2024-09-22T03:03:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.823212+0000" 2024-09-22T03:03:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.823304+0000" 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.823420+0000" 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.823566+0000" 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.824412+0000" 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.865833+0000" 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777996, 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.835832+0000" 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.835840+0000" 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.835910+0000" 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.836024+0000" 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.839416+0000" 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.872619+0000" 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.839821+0000" 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.839956+0000" 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.840060+0000" 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.840282+0000" 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.841140+0000" 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.872212+0000" 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778000, 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.866572+0000" 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.866584+0000" 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.866670+0000" 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.866723+0000" 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.866773+0000" 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.905752+0000" 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.866964+0000" 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.867069+0000" 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.867164+0000" 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.867310+0000" 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.868215+0000" 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.905403+0000" 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778002, 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.877885+0000" 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.877890+0000" 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.877990+0000" 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.878066+0000" 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.878119+0000" 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.913220+0000" 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.878314+0000" 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.878424+0000" 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.878509+0000" 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.878659+0000" 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.879553+0000" 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.912799+0000" 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778005, 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.888564+0000" 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.888568+0000" 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.888624+0000" 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.888676+0000" 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.888721+0000" 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.924740+0000" 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.888913+0000" 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.889039+0000" 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.889133+0000" 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.889289+0000" 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.890138+0000" 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.924271+0000" 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778008, 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.906171+0000" 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.906177+0000" 2024-09-22T03:03:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.906251+0000" 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.906296+0000" 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.906353+0000" 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.944002+0000" 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.906549+0000" 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.906650+0000" 2024-09-22T03:03:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.906741+0000" 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.906884+0000" 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.908218+0000" 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.943511+0000" 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778025, 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.989496+0000" 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.989502+0000" 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.989584+0000" 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.989674+0000" 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.989808+0000" 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.029934+0000" 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.993345+0000" 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.993479+0000" 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.993578+0000" 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.993746+0000" 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.994645+0000" 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.029557+0000" 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778027, 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.001303+0000" 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.001307+0000" 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.001379+0000" 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.001419+0000" 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.001464+0000" 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.059681+0000" 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.001660+0000" 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.001764+0000" 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.001855+0000" 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.002013+0000" 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.003066+0000" 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.059070+0000" 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778029, 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.018802+0000" 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.018809+0000" 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.018905+0000" 2024-09-22T03:03:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.018973+0000" 2024-09-22T03:03:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.019038+0000" 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.074347+0000" 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.019234+0000" 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.019336+0000" 2024-09-22T03:03:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.019429+0000" 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.019586+0000" 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.020461+0000" 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.073793+0000" 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778032, 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.030432+0000" 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.030437+0000" 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.030521+0000" 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.030567+0000" 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.030619+0000" 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.086293+0000" 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.030816+0000" 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.030925+0000" 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.031028+0000" 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.626 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.031174+0000" 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.032083+0000" 2024-09-22T03:03:39.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.629 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.085813+0000" 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778039, 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.073271+0000" 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.073275+0000" 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.073412+0000" 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.632 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.073587+0000" 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.073784+0000" 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.633 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.097877+0000" 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.074093+0000" 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.074552+0000" 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.074691+0000" 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.637 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.074890+0000" 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.639 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.075839+0000" 2024-09-22T03:03:39.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.641 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.097378+0000" 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778037, 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.072227+0000" 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.072235+0000" 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.643 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.072345+0000" 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.644 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.072397+0000" 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.078929+0000" 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.646 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.079305+0000" 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.079476+0000" 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.648 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.079639+0000" 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.649 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.650 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.651 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.651 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.651 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.651 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.651 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.651 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.651 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.079911+0000" 2024-09-22T03:03:39.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.652 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.081146+0000" 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777883, 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.223791+0000" 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.223799+0000" 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.223881+0000" 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.656 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.223933+0000" 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.227253+0000" 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.657 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.264078+0000" 2024-09-22T03:03:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.227452+0000" 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.227544+0000" 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.227636+0000" 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.227764+0000" 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.663 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.228431+0000" 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.664 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.263813+0000" 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777888, 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.236139+0000" 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.666 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.236146+0000" 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.236227+0000" 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.236286+0000" 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.236453+0000" 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.669 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.266094+0000" 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.236667+0000" 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.236768+0000" 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.672 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.236863+0000" 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.674 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.237012+0000" 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.675 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.241951+0000" 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.265845+0000" 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.677 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777890, 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.255579+0000" 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.678 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.255592+0000" 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.255667+0000" 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.679 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.255716+0000" 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.680 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.255969+0000" 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.681 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.282386+0000" 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.256175+0000" 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.682 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.256273+0000" 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.256391+0000" 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.684 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.256538+0000" 2024-09-22T03:03:39.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.687 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.257431+0000" 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.688 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.282142+0000" 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777894, 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.271011+0000" 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.271015+0000" 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.271092+0000" 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.271322+0000" 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.275811+0000" 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.304798+0000" 2024-09-22T03:03:39.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.276041+0000" 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.694 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.276154+0000" 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.276253+0000" 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.696 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.697 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.276398+0000" 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.277755+0000" 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.699 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.304536+0000" 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.700 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777908, 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.701 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.330758+0000" 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.330765+0000" 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.331197+0000" 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.703 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.331261+0000" 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.331443+0000" 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.704 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.367080+0000" 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.331660+0000" 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.706 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.331761+0000" 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.707 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.331861+0000" 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.332025+0000" 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.332749+0000" 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.711 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.366815+0000" 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777913, 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.352873+0000" 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.713 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.352879+0000" 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.352951+0000" 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.353025+0000" 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.715 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.353175+0000" 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.716 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.717 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.717 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.717 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.377760+0000" 2024-09-22T03:03:39.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.717 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.717 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.717 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.353392+0000" 2024-09-22T03:03:39.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.718 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.353498+0000" 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.719 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.353590+0000" 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.721 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.353733+0000" 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.722 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.357812+0000" 2024-09-22T03:03:39.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.724 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.377422+0000" 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.725 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777945, 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.527314+0000" 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.527319+0000" 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.527395+0000" 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.727 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.527463+0000" 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.527613+0000" 2024-09-22T03:03:39.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.554569+0000" 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.527819+0000" 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.730 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.527931+0000" 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.528036+0000" 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.734 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.528174+0000" 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.535619+0000" 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.736 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.554325+0000" 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.737 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777950, 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.562317+0000" 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.562320+0000" 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.562389+0000" 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.562498+0000" 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.740 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.562750+0000" 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.585599+0000" 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.562936+0000" 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.742 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.563039+0000" 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.563118+0000" 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.744 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.745 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.563257+0000" 2024-09-22T03:03:39.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.747 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.563974+0000" 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.748 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.585325+0000" 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777963, 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.647232+0000" 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.647239+0000" 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.750 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.647309+0000" 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.751 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.647362+0000" 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.647515+0000" 2024-09-22T03:03:39.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.753 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.673410+0000" 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.647720+0000" 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.647830+0000" 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.755 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.647925+0000" 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.756 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.757 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.648087+0000" 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.652198+0000" 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.760 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.672899+0000" 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.761 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777969, 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.668107+0000" 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.668111+0000" 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.668178+0000" 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.763 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.668251+0000" 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.668399+0000" 2024-09-22T03:03:39.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.700508+0000" 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.668598+0000" 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.766 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.668695+0000" 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.668784+0000" 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.769 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.770 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.668923+0000" 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.673457+0000" 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.772 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.696394+0000" 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.773 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777970, 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.672813+0000" 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.672821+0000" 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.672904+0000" 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.672959+0000" 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.677668+0000" 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.776 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.713207+0000" 2024-09-22T03:03:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.677889+0000" 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.678001+0000" 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.678147+0000" 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.780 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.781 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.678287+0000" 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.782 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.679103+0000" 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.783 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.712928+0000" 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777973, 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.693148+0000" 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.693154+0000" 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.786 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.693225+0000" 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.787 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.693289+0000" 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.696092+0000" 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.732830+0000" 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.696576+0000" 2024-09-22T03:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.790 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.696826+0000" 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.791 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.700340+0000" 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.700775+0000" 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.701477+0000" 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.795 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.732548+0000" 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777980, 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.734465+0000" 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.797 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.734469+0000" 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.734530+0000" 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.798 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.734668+0000" 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.738454+0000" 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.764172+0000" 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.738666+0000" 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.801 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.802 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.738767+0000" 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.803 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.738878+0000" 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.804 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.805 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.739035+0000" 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.740358+0000" 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.807 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.763844+0000" 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777981, 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.738959+0000" 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.809 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.738966+0000" 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.739036+0000" 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.810 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.739095+0000" 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.739372+0000" 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.785501+0000" 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.739557+0000" 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.813 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.739644+0000" 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.739731+0000" 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.816 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.739854+0000" 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.744764+0000" 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.785259+0000" 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.820 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777987, 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.775888+0000" 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.821 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.775894+0000" 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.775979+0000" 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.822 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.776027+0000" 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.823 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.776170+0000" 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.828124+0000" 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.776378+0000" 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.776472+0000" 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.826 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.776561+0000" 2024-09-22T03:03:39.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.828 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.776687+0000" 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.830 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.777650+0000" 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.827879+0000" 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.832 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777988, 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.778669+0000" 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.778675+0000" 2024-09-22T03:03:39.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.778737+0000" 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.834 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.778798+0000" 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.835 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.781766+0000" 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.836 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.828563+0000" 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.781976+0000" 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.782075+0000" 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.838 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.782166+0000" 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.839 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.840 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.782312+0000" 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.842 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.783185+0000" 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.843 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.828340+0000" 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.844 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778019, 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.954065+0000" 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.954070+0000" 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.845 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.954145+0000" 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.954197+0000" 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.954354+0000" 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.989777+0000" 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.954547+0000" 2024-09-22T03:03:39.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.849 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.954640+0000" 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.850 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.954732+0000" 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.851 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.954860+0000" 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.853 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.955641+0000" 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.989519+0000" 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.855 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778022, 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.975551+0000" 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.975557+0000" 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.856 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.975623+0000" 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.857 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.975680+0000" 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.975841+0000" 2024-09-22T03:03:39.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.859 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.859 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.859 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.859 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.859 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.859 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.859 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.017940+0000" 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.976053+0000" 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.860 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.976170+0000" 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.861 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.976279+0000" 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.862 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.976414+0000" 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.864 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:38.977187+0000" 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.017485+0000" 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "client_request", 2024-09-22T03:03:39.867 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16778036, 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.060397+0000" 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.060404+0000" 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.868 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.060502+0000" 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.060549+0000" 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.060750+0000" 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.870 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.098619+0000" 2024-09-22T03:03:39.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.061126+0000" 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.061339+0000" 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.061527+0000" 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.061782+0000" 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:03:39.876 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.063343+0000" 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.877 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: "blocking_event": { 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-09-22T03:03:39.098162+0000" 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:03:39.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time_event": { 2024-09-22T03:03:39.879 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:03:39.879 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "0.000000" 2024-09-22T03:03:39.879 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.879 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.879 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:03:39.879 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-22T03:03:39.879 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-22T03:03:39.879 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:03:40.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:40.015+0000 7fd31682c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:40.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:40.021+0000 7fd31682c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:40.183 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:40.208 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:42.209 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:03:42.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:42.361+0000 7f35a4d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:42.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:42.368+0000 7f35a4d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:42.549 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:42.642 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:42.643 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:44.644 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:03:44.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:44.779+0000 7fec9142c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:44.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:44.787+0000 7fec9142c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:44.881 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2024-09-22T03:03:44.881 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:03:44.881 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:03:45.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:45.007+0000 7f4989357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:45.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:45.019+0000 7f4989357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:45.038 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:45.093 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:45.093 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:45.298 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:03:45.299 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:03:45.314 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-22T03:03:44.344792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1336869451,"num_objects":20395,"num_object_clones":0,"num_object_copies":61185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":41160,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41105,"ondisk_log_size":41105,"up":57,"acting":57,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":418000,"kb_used_data":254132,"kb_used_omap":0,"kb_used_meta":163776,"kb_avail":377069360,"statfs":{"total":386547056640,"available":386119024640,"internally_reserved":0,"allocated":260231168,"data_stored":4141637872,"data_compressed":196650015,"data_compressed_allocated":258834432,"data_compressed_original":4141105152,"omap_allocated":0,"internal_metadata":167706624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-136839168,"num_objects":-2088,"num_object_clones":0,"num_object_copies":-6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-21153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002146"},"pg_stats":[{"pgid":"3.f","version":"41'2584","reported_seq":2702,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-22T03:03:42.162290+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:03:42.162290+0000","last_peered":"2024-09-22T03:03:42.162290+0000","last_clean":"2024-09-22T03:03:42.162290+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:03:42.162290+0000","last_undegraded":"2024-09-22T03:03:42.162290+0000","last_fullsized":"2024-09-22T03:03:42.162290+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2584,"log_dups_size":0,"ondisk_log_size":2584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85000192,"num_objects":1297,"num_object_clones":0,"num_object_copies":3891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2585,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"41'2566","reported_seq":2664,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:43.731876+0000","last_change":"2024-09-22T03:03:35.275552+0000","last_active":"2024-09-22T03:03:43.731876+0000","last_peered":"2024-09-22T03:03:43.731876+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:03:35.275552+0000","last_became_peered":"2024-09-22T03:03:35.275552+0000","last_unstale":"2024-09-22T03:03:43.731876+0000","last_undegraded":"2024-09-22T03:03:35.204523+0000","last_fullsized":"2024-09-22T03:03:35.203672+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2566,"log_dups_size":0,"ondisk_log_size":2566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82509824,"num_objects":1259,"num_object_clones":0,"num_object_copies":3777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1259}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"41'2616","reported_seq":2717,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-22T03:03:43.794082+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:03:43.794082+0000","last_peered":"2024-09-22T03:03:43.794082+0000","last_clean":"2024-09-22T03:03:43.794082+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:03:43.794082+0000","last_undegraded":"2024-09-22T03:03:43.794082+0000","last_fullsized":"2024-09-22T03:03:43.794082+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2616,"log_dups_size":0,"ondisk_log_size":2616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81133592,"num_objects":1239,"num_object_clones":0,"num_object_copies":3717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2617,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"41'2482","reported_seq":2576,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:42.185645+0000","last_change":"2024-09-22T03:03:35.229958+0000","last_active":"2024-09-22T03:03:42.185645+0000","last_peered":"2024-09-22T03:03:42.185645+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:03:35.229958+0000","last_became_peered":"2024-09-22T03:03:35.229958+0000","last_unstale":"2024-09-22T03:03:42.185645+0000","last_undegraded":"2024-09-22T03:03:35.156515+0000","last_fullsized":"2024-09-22T03:03:35.155923+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2482,"log_dups_size":0,"ondisk_log_size":2482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79364096,"num_objects":1211,"num_object_clones":0,"num_object_copies":3633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"41'2495","reported_seq":2585,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:39.932339+0000","last_change":"2024-09-22T03:03:35.263083+0000","last_active":"2024-09-22T03:03:39.932339+0000","last_peered":"2024-09-22T03:03:39.932339+0000","last_clean":"2024-09-22T03:02:34.806110+0000","last_became_active":"2024-09-22T03:03:35.213083+0000","last_became_peered":"2024-09-22T03:03:35.213083+0000","last_unstale":"2024-09-22T03:03:39.932339+0000","last_undegraded":"2024-09-22T03:03:35.133115+0000","last_fullsized":"2024-09-22T03:03:35.132066+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2495,"log_dups_size":0,"ondisk_log_size":2495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86114304,"num_objects":1314,"num_object_clones":0,"num_object_copies":3942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2496,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"41'2549","reported_seq":2643,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:42.269973+0000","last_change":"2024-09-22T03:03:14.492811+0000","last_active":"2024-09-22T03:03:42.269973+0000","last_peered":"2024-09-22T03:03:42.269973+0000","last_clean":"2024-09-22T03:03:02.467151+0000","last_became_active":"2024-09-22T03:03:14.492811+0000","last_became_peered":"2024-09-22T03:03:14.492811+0000","last_unstale":"2024-09-22T03:03:42.269973+0000","last_undegraded":"2024-09-22T03:03:14.432326+0000","last_fullsized":"2024-09-22T03:03:14.431619+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2549,"log_dups_size":0,"ondisk_log_size":2549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81264640,"num_objects":1240,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1240}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"41'2513","reported_seq":2605,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:42.263716+0000","last_change":"2024-09-22T03:03:14.472441+0000","last_active":"2024-09-22T03:03:42.263716+0000","last_peered":"2024-09-22T03:03:42.263716+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:03:14.472441+0000","last_became_peered":"2024-09-22T03:03:14.472441+0000","last_unstale":"2024-09-22T03:03:42.263716+0000","last_undegraded":"2024-09-22T03:03:14.432933+0000","last_fullsized":"2024-09-22T03:03:14.431951+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2513,"log_dups_size":0,"ondisk_log_size":2513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":1216,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2514,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1216}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"41'2613","reported_seq":2707,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:43.665357+0000","last_change":"2024-09-22T03:03:35.433745+0000","last_active":"2024-09-22T03:03:43.665357+0000","last_peered":"2024-09-22T03:03:43.665357+0000","last_clean":"2024-09-22T03:02:34.917574+0000","last_became_active":"2024-09-22T03:03:35.279445+0000","last_became_peered":"2024-09-22T03:03:35.279445+0000","last_unstale":"2024-09-22T03:03:43.665357+0000","last_undegraded":"2024-09-22T03:03:35.200545+0000","last_fullsized":"2024-09-22T03:03:35.155977+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2613,"log_dups_size":0,"ondisk_log_size":2613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83099648,"num_objects":1268,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2614,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1268}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-09-22T03:03:35.221639+0000","last_change":"2024-09-22T03:03:35.221272+0000","last_active":"2024-09-22T03:03:35.221639+0000","last_peered":"2024-09-22T03:03:35.221639+0000","last_clean":"2024-09-22T03:01:39.208012+0000","last_became_active":"2024-09-22T03:03:35.221272+0000","last_became_peered":"2024-09-22T03:03:35.221272+0000","last_unstale":"2024-09-22T03:03:35.221639+0000","last_undegraded":"2024-09-22T03:03:35.221639+0000","last_fullsized":"2024-09-22T03:03:35.138294+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"41'2638","reported_seq":2740,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-22T03:03:43.681471+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:03:43.681471+0000","last_peered":"2024-09-22T03:03:43.681471+0000","last_clean":"2024-09-22T03:03:43.681471+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:03:43.681471+0000","last_undegraded":"2024-09-22T03:03:43.681471+0000","last_fullsized":"2024-09-22T03:03:43.681471+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2638,"log_dups_size":0,"ondisk_log_size":2638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80412672,"num_objects":1227,"num_object_clones":0,"num_object_copies":3681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2639,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":101,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-09-22T03:03:35.125028+0000","last_change":"2024-09-22T03:03:14.502104+0000","last_active":"2024-09-22T03:03:35.125028+0000","last_peered":"2024-09-22T03:03:35.125028+0000","last_clean":"2024-09-22T03:03:02.409542+0000","last_became_active":"2024-09-22T03:03:14.502104+0000","last_became_peered":"2024-09-22T03:03:14.502104+0000","last_unstale":"2024-09-22T03:03:35.125028+0000","last_undegraded":"2024-09-22T03:03:35.125028+0000","last_fullsized":"2024-09-22T03:03:14.423546+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"41'2622","reported_seq":2713,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:42.254807+0000","last_change":"2024-09-22T03:03:35.279864+0000","last_active":"2024-09-22T03:03:42.254807+0000","last_peered":"2024-09-22T03:03:42.254807+0000","last_clean":"2024-09-22T03:02:34.949467+0000","last_became_active":"2024-09-22T03:03:35.218911+0000","last_became_peered":"2024-09-22T03:03:35.218911+0000","last_unstale":"2024-09-22T03:03:42.254807+0000","last_undegraded":"2024-09-22T03:03:35.151228+0000","last_fullsized":"2024-09-22T03:03:35.125952+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2622,"log_dups_size":0,"ondisk_log_size":2622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84869120,"num_objects":1295,"num_object_clones":0,"num_object_copies":3885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2623,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1295}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":26,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:35.287263+0000","last_change":"2024-09-22T03:03:35.286903+0000","last_active":"2024-09-22T03:03:35.287263+0000","last_peered":"2024-09-22T03:03:35.287263+0000","last_clean":"2024-09-22T02:58:47.350108+0000","last_became_active":"2024-09-22T03:03:35.286903+0000","last_became_peered":"2024-09-22T03:03:35.286903+0000","last_unstale":"2024-09-22T03:03:35.287263+0000","last_undegraded":"2024-09-22T03:03:35.134584+0000","last_fullsized":"2024-09-22T03:03:35.133875+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"41'2505","reported_seq":2607,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:39.854873+0000","last_change":"2024-09-22T03:03:35.383572+0000","last_active":"2024-09-22T03:03:39.854873+0000","last_peered":"2024-09-22T03:03:39.854873+0000","last_clean":"2024-09-22T03:02:34.905097+0000","last_became_active":"2024-09-22T03:03:35.305442+0000","last_became_peered":"2024-09-22T03:03:35.305442+0000","last_unstale":"2024-09-22T03:03:39.854873+0000","last_undegraded":"2024-09-22T03:03:35.138373+0000","last_fullsized":"2024-09-22T03:03:35.137164+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2505,"log_dups_size":0,"ondisk_log_size":2505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82837504,"num_objects":1264,"num_object_clones":0,"num_object_copies":3792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2506,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":85,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-22T03:03:35.300655+0000","last_change":"2024-09-22T03:03:35.301148+0000","last_active":"2024-09-22T03:03:35.300655+0000","last_peered":"2024-09-22T03:03:35.300655+0000","last_clean":"2024-09-22T02:58:44.315779+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:03:35.300655+0000","last_undegraded":"2024-09-22T03:03:35.300655+0000","last_fullsized":"2024-09-22T03:03:35.300655+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":225,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:35.155522+0000","last_change":"2024-09-22T03:03:14.492627+0000","last_active":"2024-09-22T03:03:35.155522+0000","last_peered":"2024-09-22T03:03:35.155522+0000","last_clean":"2024-09-22T03:03:02.414940+0000","last_became_active":"2024-09-22T03:03:14.492627+0000","last_became_peered":"2024-09-22T03:03:14.492627+0000","last_unstale":"2024-09-22T03:03:35.155522+0000","last_undegraded":"2024-09-22T03:03:14.436705+0000","last_fullsized":"2024-09-22T03:03:14.435801+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"41'2712","reported_seq":2813,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-22T03:03:39.965369+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:03:39.965369+0000","last_peered":"2024-09-22T03:03:39.965369+0000","last_clean":"2024-09-22T03:03:39.965369+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:03:39.965369+0000","last_undegraded":"2024-09-22T03:03:39.965369+0000","last_fullsized":"2024-09-22T03:03:39.965369+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2712,"log_dups_size":0,"ondisk_log_size":2712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88145920,"num_objects":1345,"num_object_clones":0,"num_object_copies":4035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2713,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"41'2584","reported_seq":2667,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:39.798846+0000","last_change":"2024-09-22T03:03:35.394618+0000","last_active":"2024-09-22T03:03:39.798846+0000","last_peered":"2024-09-22T03:03:39.798846+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:03:35.304901+0000","last_became_peered":"2024-09-22T03:03:35.304901+0000","last_unstale":"2024-09-22T03:03:39.798846+0000","last_undegraded":"2024-09-22T03:03:35.137444+0000","last_fullsized":"2024-09-22T03:03:35.136339+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2584,"log_dups_size":0,"ondisk_log_size":2584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87752704,"num_objects":1339,"num_object_clones":0,"num_object_copies":4017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2585,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-09-22T03:03:35.184385+0000","last_change":"2024-09-22T03:03:14.507004+0000","last_active":"2024-09-22T03:03:35.184385+0000","last_peered":"2024-09-22T03:03:35.184385+0000","last_clean":"2024-09-22T03:03:02.417939+0000","last_became_active":"2024-09-22T03:03:14.507004+0000","last_became_peered":"2024-09-22T03:03:14.507004+0000","last_unstale":"2024-09-22T03:03:35.184385+0000","last_undegraded":"2024-09-22T03:03:35.184385+0000","last_fullsized":"2024-09-22T03:03:14.432930+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"41'2593","reported_seq":2688,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-22T03:03:39.872516+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:03:39.872516+0000","last_peered":"2024-09-22T03:03:39.872516+0000","last_clean":"2024-09-22T03:03:39.872516+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:03:39.872516+0000","last_undegraded":"2024-09-22T03:03:39.872516+0000","last_fullsized":"2024-09-22T03:03:39.872516+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2593,"log_dups_size":0,"ondisk_log_size":2593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90570752,"num_objects":1382,"num_object_clones":0,"num_object_copies":4146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2594,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-09-22T03:03:35.222452+0000","last_change":"2024-09-22T03:03:35.221953+0000","last_active":"2024-09-22T03:03:35.222452+0000","last_peered":"2024-09-22T03:03:35.222452+0000","last_clean":"2024-09-22T03:01:03.231788+0000","last_became_active":"2024-09-22T03:03:35.221953+0000","last_became_peered":"2024-09-22T03:03:35.221953+0000","last_unstale":"2024-09-22T03:03:35.222452+0000","last_undegraded":"2024-09-22T03:03:35.222452+0000","last_fullsized":"2024-09-22T03:03:35.153597+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"41'2490","reported_seq":2569,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:39.806674+0000","last_change":"2024-09-22T03:03:35.274472+0000","last_active":"2024-09-22T03:03:39.806674+0000","last_peered":"2024-09-22T03:03:39.806674+0000","last_clean":"2024-09-22T03:02:34.870768+0000","last_became_active":"2024-09-22T03:03:35.215913+0000","last_became_peered":"2024-09-22T03:03:35.215913+0000","last_unstale":"2024-09-22T03:03:39.806674+0000","last_undegraded":"2024-09-22T03:03:35.126935+0000","last_fullsized":"2024-09-22T03:03:35.125668+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2490,"log_dups_size":0,"ondisk_log_size":2490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84869120,"num_objects":1295,"num_object_clones":0,"num_object_copies":3885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2491,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":62,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-09-22T03:03:35.285453+0000","last_change":"2024-09-22T03:03:35.284878+0000","last_active":"2024-09-22T03:03:35.285453+0000","last_peered":"2024-09-22T03:03:35.285453+0000","last_clean":"2024-09-22T03:02:15.309567+0000","last_became_active":"2024-09-22T03:03:35.284878+0000","last_became_peered":"2024-09-22T03:03:35.284878+0000","last_unstale":"2024-09-22T03:03:35.285453+0000","last_undegraded":"2024-09-22T03:03:35.285453+0000","last_fullsized":"2024-09-22T03:03:35.201819+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"41'2443","reported_seq":2527,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:03:43.476069+0000","last_change":"2024-09-22T03:03:35.229724+0000","last_active":"2024-09-22T03:03:43.476069+0000","last_peered":"2024-09-22T03:03:43.476069+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:03:35.229724+0000","last_became_peered":"2024-09-22T03:03:35.229724+0000","last_unstale":"2024-09-22T03:03:43.476069+0000","last_undegraded":"2024-09-22T03:03:35.151755+0000","last_fullsized":"2024-09-22T03:03:35.150944+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2443,"log_dups_size":0,"ondisk_log_size":2443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78643200,"num_objects":1200,"num_object_clones":0,"num_object_copies":3600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2444,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1200}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-22T03:03:35.308138+0000","last_change":"2024-09-22T03:03:35.308817+0000","last_active":"2024-09-22T03:03:35.308138+0000","last_peered":"2024-09-22T03:03:35.308138+0000","last_clean":"2024-09-22T03:02:15.256754+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:03:35.308138+0000","last_undegraded":"2024-09-22T03:03:35.308138+0000","last_fullsized":"2024-09-22T03:03:35.308138+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1336279064,"num_objects":20391,"num_object_clones":0,"num_object_copies":61173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13901,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":41021,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":256520192,"data_stored":4104126536,"data_compressed":194885882,"data_compressed_allocated":256507904,"data_compressed_original":4104126464,"omap_allocated":0,"internal_metadata":0},"log_size":41005,"ondisk_log_size":41005,"up":37,"acting":37,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":18,"acting":18,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691845,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":120028,"kb_used_data":75068,"kb_used_omap":0,"kb_used_meta":44928,"kb_avail":94251812,"statfs":{"total":96636764160,"available":96513855488,"internally_reserved":0,"allocated":76869632,"data_stored":1224469258,"data_compressed":58138383,"data_compressed_allocated":76521472,"data_compressed_original":1224343552,"omap_allocated":0,"internal_metadata":46006272},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":26,"seq":111669149795,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96196,"kb_used_data":61684,"kb_used_omap":0,"kb_used_meta":34496,"kb_avail":94275644,"statfs":{"total":96636764160,"available":96538259456,"internally_reserved":0,"allocated":63164416,"data_stored":1005624229,"data_compressed":47749366,"data_compressed_allocated":62849024,"data_compressed_original":1005502464,"omap_allocated":0,"internal_metadata":35323904},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":110012,"kb_used_data":56812,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94261828,"statfs":{"total":96636764160,"available":96524111872,"internally_reserved":0,"allocated":58175488,"data_stored":925035799,"data_compressed":43921605,"data_compressed_allocated":57810944,"data_compressed_original":924893184,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888153,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91764,"kb_used_data":60568,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94280076,"statfs":{"total":96636764160,"available":96542797824,"internally_reserved":0,"allocated":62021632,"data_stored":986508586,"data_compressed":46840661,"data_compressed_allocated":61652992,"data_compressed_original":986365952,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57114624,"data_stored":913768472,"data_compressed":43390614,"data_compressed_allocated":57110528,"data_compressed_original":913768448,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61034496,"data_stored":976486424,"data_compressed":46368798,"data_compressed_allocated":61030400,"data_compressed_original":976486400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62808064,"data_stored":1004929024,"data_compressed":47719407,"data_compressed_allocated":62808064,"data_compressed_original":1004929024,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75563008,"data_stored":1208942616,"data_compressed":57407063,"data_compressed_allocated":75558912,"data_compressed_original":1208942592,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:03:47.094 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:03:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:47.243+0000 7f20e8d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:47.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:47.250+0000 7f20e8d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:47.423 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:47.471 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:49.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:03:49.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:49.619+0000 7f97ad103640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:49.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:49.626+0000 7f97ad103640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:49.853 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:49.927 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:49.927 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:50.316 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2024-09-22T03:03:50.317 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:03:50.317 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-22T03:03:50.317 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:03:50.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:03:50.319 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:03:50.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:03:50.434 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:03:50.438 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:03:50.438 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:03:51.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:03:52.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:52.063+0000 7fc7c590e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:52.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:52.069+0000 7fc7c590e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:52.323 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:52.370 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:54.371 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:03:54.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:54.510+0000 7f98dfe23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:54.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:54.517+0000 7f98dfe23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:54.633 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:54.743 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:55.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:03:55.559 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:03:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:03:55.562 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:03:56.744 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:03:56.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:56.863+0000 7f1ed0063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:56.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:56.869+0000 7f1ed0063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:56.930 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:56.993 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:56.993 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:03:58.994 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:03:59.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:59.111+0000 7fd6aad97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:59.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:03:59.117+0000 7fd6aad97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:03:59.241 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:03:59.453 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:03:59.454 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:04:00.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:04:00.676 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:04:00.679 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:04:00.679 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:04:01.455 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:01.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:01.598+0000 7faae8197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:01.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:01.606+0000 7faae8197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:01.858 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:04:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:04:02.035 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:04:04.036 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:04.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:04.157+0000 7fb308ea9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:04.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:04.165+0000 7fb308ea9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:05.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:04:05.870 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-09-22T03:04:05.870 INFO:teuthology.orchestra.run.smithi190.stdout: "ops_in_flight": [ 2024-09-22T03:04:05.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:04:05.871 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "replicated_request", 2024-09-22T03:04:05.871 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777362, 2024-09-22T03:04:05.871 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:04:05.871 INFO:teuthology.orchestra.run.smithi190.stdout: "RepRequest": { 2024-09-22T03:04:05.871 INFO:teuthology.orchestra.run.smithi190.stdout: "reqid": "client.4243.0:46765", 2024-09-22T03:04:05.871 INFO:teuthology.orchestra.run.smithi190.stdout: "pgid": "3.c", 2024-09-22T03:04:05.871 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 44, 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: "min_epoch": 43, 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: "oid": "3:3804bd0d:::benchmark_data_smithi081_30037_object16057:head", 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: "from": "0" 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "replicated_request", 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 135, 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: "RepRequest": { 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: "reqid": "client.4243.0:46760", 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: "pgid": "3.6", 2024-09-22T03:04:05.872 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 44, 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: "min_epoch": 43, 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: "oid": "3:63f3a918:::benchmark_data_smithi081_30037_object16052:head", 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: "from": "1" 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "background_recovery_sub", 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777361, 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": {} 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "replicated_request", 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 16777359, 2024-09-22T03:04:05.873 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: "RepRequest": { 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: "reqid": "client.4243.0:46758", 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: "pgid": "3.9", 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 44, 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: "min_epoch": 43, 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: "oid": "3:91003fa3:::benchmark_data_smithi081_30037_object16050:head", 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: "from": "0" 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "replicated_request", 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 134, 2024-09-22T03:04:05.874 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: "RepRequest": { 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: "reqid": "client.4243.0:46757", 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: "pgid": "3.5", 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 44, 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: "min_epoch": 43, 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: "oid": "3:ac092561:::benchmark_data_smithi081_30037_object16049:head", 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: "from": "1" 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "replicated_request", 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: "id": 33554532, 2024-09-22T03:04:05.875 INFO:teuthology.orchestra.run.smithi190.stdout: "detail": { 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout: "RepRequest": { 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout: "reqid": "client.4243.0:46761", 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout: "pgid": "3.b", 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 44, 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout: "min_epoch": 43, 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout: "oid": "3:d3fc07e6:::benchmark_data_smithi081_30037_object16053:head", 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout: "from": "3" 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-09-22T03:04:05.876 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-09-22T03:04:06.392 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:06.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:06.541+0000 7fc5d7997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:06.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:06.549+0000 7fc5d7997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:08.842 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:08.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:08.981+0000 7f3de4357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:08.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:08.991+0000 7f3de4357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:10.884 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 2] 2024-09-22T03:04:10.884 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:04:10.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:11.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:11.029+0000 7f74a6797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:11.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:11.041+0000 7f74a6797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:11.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:11.389 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:11.389 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:11.408 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-22T03:04:10.350117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":915341899,"num_objects":13963,"num_object_clones":0,"num_object_copies":41889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":47592,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47539,"ondisk_log_size":47539,"up":75,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":359540,"kb_used_data":190628,"kb_used_omap":0,"kb_used_meta":168832,"kb_avail":377127820,"statfs":{"total":386547056640,"available":386178887680,"internally_reserved":0,"allocated":195203072,"data_stored":3098895581,"data_compressed":147131874,"data_compressed_allocated":193658880,"data_compressed_original":3098296320,"omap_allocated":0,"internal_metadata":172883968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-98500608,"num_objects":-1503,"num_object_clones":0,"num_object_copies":-4509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-7227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1503,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.889590"},"pg_stats":[{"pgid":"3.f","version":"44'2957","reported_seq":3084,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:07.315606+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:07.315606+0000","last_peered":"2024-09-22T03:04:07.315606+0000","last_clean":"2024-09-22T03:04:07.315606+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:07.315606+0000","last_undegraded":"2024-09-22T03:04:07.315606+0000","last_fullsized":"2024-09-22T03:04:07.315606+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2957,"log_dups_size":0,"ondisk_log_size":2957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60555264,"num_objects":924,"num_object_clones":0,"num_object_copies":2772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2958,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"44'2955","reported_seq":3069,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:08.748975+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:08.748975+0000","last_peered":"2024-09-22T03:04:08.748975+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:08.748975+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2955,"log_dups_size":0,"ondisk_log_size":2955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57016320,"num_objects":870,"num_object_clones":0,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2956,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":870}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"44'3001","reported_seq":3111,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:08.717982+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:08.717982+0000","last_peered":"2024-09-22T03:04:08.717982+0000","last_clean":"2024-09-22T03:04:08.717982+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:08.717982+0000","last_undegraded":"2024-09-22T03:04:08.717982+0000","last_fullsized":"2024-09-22T03:04:08.717982+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3001,"log_dups_size":0,"ondisk_log_size":3001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55902232,"num_objects":854,"num_object_clones":0,"num_object_copies":2562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3002,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"44'2837","reported_seq":2941,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:07.135119+0000","last_change":"2024-09-22T03:04:05.652014+0000","last_active":"2024-09-22T03:04:07.135119+0000","last_peered":"2024-09-22T03:04:07.135119+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:07.135119+0000","last_undegraded":"2024-09-22T03:04:05.225991+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2837,"log_dups_size":0,"ondisk_log_size":2837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56098816,"num_objects":856,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2838,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":856}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"44'2951","reported_seq":3055,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:09.970949+0000","last_change":"2024-09-22T03:04:05.620254+0000","last_active":"2024-09-22T03:04:09.970949+0000","last_peered":"2024-09-22T03:04:09.970949+0000","last_clean":"2024-09-22T03:02:34.806110+0000","last_became_active":"2024-09-22T03:04:05.614390+0000","last_became_peered":"2024-09-22T03:04:05.614390+0000","last_unstale":"2024-09-22T03:04:09.970949+0000","last_undegraded":"2024-09-22T03:04:05.275848+0000","last_fullsized":"2024-09-22T03:04:05.120934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2951,"log_dups_size":0,"ondisk_log_size":2951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56229888,"num_objects":858,"num_object_clones":0,"num_object_copies":2574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":471,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2952,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"44'2908","reported_seq":3094,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:07.320655+0000","last_change":"2024-09-22T03:04:05.766323+0000","last_active":"2024-09-22T03:04:07.320655+0000","last_peered":"2024-09-22T03:04:07.320655+0000","last_clean":"2024-09-22T03:03:02.467151+0000","last_became_active":"2024-09-22T03:04:05.627124+0000","last_became_peered":"2024-09-22T03:04:05.627124+0000","last_unstale":"2024-09-22T03:04:07.320655+0000","last_undegraded":"2024-09-22T03:04:07.320655+0000","last_fullsized":"2024-09-22T03:04:05.261855+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2908,"log_dups_size":0,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57737216,"num_objects":881,"num_object_clones":0,"num_object_copies":2643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2909,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":881}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"44'2869","reported_seq":3057,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:07.340634+0000","last_change":"2024-09-22T03:04:05.723254+0000","last_active":"2024-09-22T03:04:07.340634+0000","last_peered":"2024-09-22T03:04:07.340634+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:05.643757+0000","last_became_peered":"2024-09-22T03:04:05.643757+0000","last_unstale":"2024-09-22T03:04:07.340634+0000","last_undegraded":"2024-09-22T03:04:07.340634+0000","last_fullsized":"2024-09-22T03:04:05.261970+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2869,"log_dups_size":0,"ondisk_log_size":2869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56426496,"num_objects":861,"num_object_clones":0,"num_object_copies":2583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":861}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"44'3011","reported_seq":3279,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:08.816541+0000","last_change":"2024-09-22T03:04:08.668820+0000","last_active":"2024-09-22T03:04:08.816541+0000","last_peered":"2024-09-22T03:04:08.816541+0000","last_clean":"2024-09-22T03:02:34.917574+0000","last_became_active":"2024-09-22T03:04:05.615659+0000","last_became_peered":"2024-09-22T03:04:05.615659+0000","last_unstale":"2024-09-22T03:04:08.816541+0000","last_undegraded":"2024-09-22T03:04:08.816541+0000","last_fullsized":"2024-09-22T03:04:05.160183+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3011,"log_dups_size":0,"ondisk_log_size":3011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57081856,"num_objects":871,"num_object_clones":0,"num_object_copies":2613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3011,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":871}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":85,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:05.117787+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:05.117787+0000","last_peered":"2024-09-22T03:04:05.117787+0000","last_clean":"2024-09-22T03:04:05.117787+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:05.117787+0000","last_undegraded":"2024-09-22T03:04:05.117787+0000","last_fullsized":"2024-09-22T03:04:05.117787+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"44'3014","reported_seq":3125,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:08.750089+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:08.750089+0000","last_peered":"2024-09-22T03:04:08.750089+0000","last_clean":"2024-09-22T03:04:08.750089+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:08.750089+0000","last_undegraded":"2024-09-22T03:04:08.750089+0000","last_fullsized":"2024-09-22T03:04:08.750089+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3014,"log_dups_size":0,"ondisk_log_size":3014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55771136,"num_objects":851,"num_object_clones":0,"num_object_copies":2553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3015,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":114,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:05.135246+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:05.135246+0000","last_peered":"2024-09-22T03:04:05.135246+0000","last_clean":"2024-09-22T03:04:05.135246+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:05.135246+0000","last_undegraded":"2024-09-22T03:04:05.135246+0000","last_fullsized":"2024-09-22T03:04:05.135246+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"44'3019","reported_seq":3213,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:07.326275+0000","last_change":"2024-09-22T03:04:06.795404+0000","last_active":"2024-09-22T03:04:07.326275+0000","last_peered":"2024-09-22T03:04:07.326275+0000","last_clean":"2024-09-22T03:02:34.949467+0000","last_became_active":"2024-09-22T03:04:05.611020+0000","last_became_peered":"2024-09-22T03:04:05.611020+0000","last_unstale":"2024-09-22T03:04:07.326275+0000","last_undegraded":"2024-09-22T03:04:07.326275+0000","last_fullsized":"2024-09-22T03:04:05.189813+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3019,"log_dups_size":0,"ondisk_log_size":3019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58851328,"num_objects":898,"num_object_clones":0,"num_object_copies":2694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":898}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":100,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:05.119260+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:05.119260+0000","last_peered":"2024-09-22T03:04:05.119260+0000","last_clean":"2024-09-22T03:04:05.119260+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:05.119260+0000","last_undegraded":"2024-09-22T03:04:05.119260+0000","last_fullsized":"2024-09-22T03:04:05.119260+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"44'2923","reported_seq":3035,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:09.980883+0000","last_change":"2024-09-22T03:04:05.617923+0000","last_active":"2024-09-22T03:04:09.980883+0000","last_peered":"2024-09-22T03:04:09.980883+0000","last_clean":"2024-09-22T03:02:34.905097+0000","last_became_active":"2024-09-22T03:04:05.614727+0000","last_became_peered":"2024-09-22T03:04:05.614727+0000","last_unstale":"2024-09-22T03:04:09.980883+0000","last_undegraded":"2024-09-22T03:04:05.160798+0000","last_fullsized":"2024-09-22T03:04:05.116295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55443456,"num_objects":846,"num_object_clones":0,"num_object_copies":2538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":95,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:05.118238+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:05.118238+0000","last_peered":"2024-09-22T03:04:05.118238+0000","last_clean":"2024-09-22T03:04:05.118238+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:05.118238+0000","last_undegraded":"2024-09-22T03:04:05.118238+0000","last_fullsized":"2024-09-22T03:04:05.118238+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":237,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:05.191082+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:05.191082+0000","last_peered":"2024-09-22T03:04:05.191082+0000","last_clean":"2024-09-22T03:04:05.191082+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:05.191082+0000","last_undegraded":"2024-09-22T03:04:05.191082+0000","last_fullsized":"2024-09-22T03:04:05.191082+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"44'3199","reported_seq":3309,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:09.884720+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:09.884720+0000","last_peered":"2024-09-22T03:04:09.884720+0000","last_clean":"2024-09-22T03:04:09.884720+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:09.884720+0000","last_undegraded":"2024-09-22T03:04:09.884720+0000","last_fullsized":"2024-09-22T03:04:09.884720+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56229888,"num_objects":858,"num_object_clones":0,"num_object_copies":2574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3200,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"44'3015","reported_seq":3107,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:08.675962+0000","last_change":"2024-09-22T03:04:05.634562+0000","last_active":"2024-09-22T03:04:08.675962+0000","last_peered":"2024-09-22T03:04:08.675962+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:08.675962+0000","last_undegraded":"2024-09-22T03:04:05.200849+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3015,"log_dups_size":0,"ondisk_log_size":3015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59506688,"num_objects":908,"num_object_clones":0,"num_object_copies":2724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3016,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":908}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:05.201798+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:05.201798+0000","last_peered":"2024-09-22T03:04:05.201798+0000","last_clean":"2024-09-22T03:04:05.201798+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:05.201798+0000","last_undegraded":"2024-09-22T03:04:05.201798+0000","last_fullsized":"2024-09-22T03:04:05.201798+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"44'3077","reported_seq":3181,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:09.800707+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:09.800707+0000","last_peered":"2024-09-22T03:04:09.800707+0000","last_clean":"2024-09-22T03:04:09.800707+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:09.800707+0000","last_undegraded":"2024-09-22T03:04:09.800707+0000","last_fullsized":"2024-09-22T03:04:09.800707+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3077,"log_dups_size":0,"ondisk_log_size":3077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58851328,"num_objects":898,"num_object_clones":0,"num_object_copies":2694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:05.114061+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:05.114061+0000","last_peered":"2024-09-22T03:04:05.114061+0000","last_clean":"2024-09-22T03:04:05.114061+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:05.114061+0000","last_undegraded":"2024-09-22T03:04:05.114061+0000","last_fullsized":"2024-09-22T03:04:05.114061+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"44'2892","reported_seq":2980,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:07.169877+0000","last_change":"2024-09-22T03:04:05.626371+0000","last_active":"2024-09-22T03:04:07.169877+0000","last_peered":"2024-09-22T03:04:07.169877+0000","last_clean":"2024-09-22T03:02:34.870768+0000","last_became_active":"2024-09-22T03:04:05.621831+0000","last_became_peered":"2024-09-22T03:04:05.621831+0000","last_unstale":"2024-09-22T03:04:07.169877+0000","last_undegraded":"2024-09-22T03:04:05.186047+0000","last_fullsized":"2024-09-22T03:04:05.135956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2892,"log_dups_size":0,"ondisk_log_size":2892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58523648,"num_objects":893,"num_object_clones":0,"num_object_copies":2679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":455,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2893,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":893}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:05.364750+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:05.364750+0000","last_peered":"2024-09-22T03:04:05.364750+0000","last_clean":"2024-09-22T03:04:05.364750+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:05.364750+0000","last_undegraded":"2024-09-22T03:04:05.364750+0000","last_fullsized":"2024-09-22T03:04:05.364750+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"44'2811","reported_seq":2912,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:08.726183+0000","last_change":"2024-09-22T03:04:05.673295+0000","last_active":"2024-09-22T03:04:08.726183+0000","last_peered":"2024-09-22T03:04:08.726183+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:08.726183+0000","last_undegraded":"2024-09-22T03:04:05.671962+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":2811,"log_dups_size":0,"ondisk_log_size":2811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":832,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2812,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":832}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-22T03:04:05.363869+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:05.363869+0000","last_peered":"2024-09-22T03:04:05.363869+0000","last_clean":"2024-09-22T03:04:05.363869+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:05.363869+0000","last_undegraded":"2024-09-22T03:04:05.363869+0000","last_fullsized":"2024-09-22T03:04:05.363869+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":914751512,"num_objects":13959,"num_object_clones":0,"num_object_copies":41877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":47453,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194912256,"data_stored":3118399560,"data_compressed":148078293,"data_compressed_allocated":194899968,"data_compressed_original":3118399488,"omap_allocated":0,"internal_metadata":0},"log_size":47439,"ondisk_log_size":47439,"up":48,"acting":37,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89756,"kb_used_data":48972,"kb_used_omap":0,"kb_used_meta":40768,"kb_avail":94282084,"statfs":{"total":96636764160,"available":96544854016,"internally_reserved":0,"allocated":50147328,"data_stored":796530602,"data_compressed":37817024,"data_compressed_allocated":49774592,"data_compressed_original":796393472,"omap_allocated":0,"internal_metadata":41746432},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":98312,"kb_used_data":58740,"kb_used_omap":0,"kb_used_meta":39552,"kb_avail":94273528,"statfs":{"total":96636764160,"available":96536092672,"internally_reserved":0,"allocated":60149760,"data_stored":956242354,"data_compressed":45402918,"data_compressed_allocated":59760640,"data_compressed_original":956088320,"omap_allocated":0,"internal_metadata":40501248},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673284,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":93452,"kb_used_data":40248,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94278388,"statfs":{"total":96636764160,"available":96541069312,"internally_reserved":0,"allocated":41213952,"data_stored":653269431,"data_compressed":31016142,"data_compressed_allocated":40824832,"data_compressed_original":653115392,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78020,"kb_used_data":42668,"kb_used_omap":0,"kb_used_meta":35328,"kb_avail":94293820,"statfs":{"total":96636764160,"available":96556871680,"internally_reserved":0,"allocated":43692032,"data_stored":692853194,"data_compressed":32895790,"data_compressed_allocated":43298816,"data_compressed_original":692699136,"omap_allocated":0,"internal_metadata":36175872},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40255488,"data_stored":644022296,"data_compressed":30581623,"data_compressed_allocated":40251392,"data_compressed_original":644022272,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42790912,"data_stored":684589080,"data_compressed":32507951,"data_compressed_allocated":42786816,"data_compressed_original":684589056,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62808064,"data_stored":1004929024,"data_compressed":47719407,"data_compressed_allocated":62808064,"data_compressed_original":1004929024,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49057792,"data_stored":784859160,"data_compressed":37269312,"data_compressed_allocated":49053696,"data_compressed_original":784859136,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:11.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:11.456+0000 7f230f997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:11.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:11.462+0000 7f230f997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:13.737 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:13.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:13.865+0000 7f133eca9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:13.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:13.871+0000 7f133eca9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:16.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:16.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:16.357+0000 7fa7e970e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:16.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:16.365+0000 7fa7e970e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:16.411 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 2] 2024-09-22T03:04:16.411 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:04:16.411 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:04:16.411 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:04:16.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:16.554+0000 7fda16ea9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:16.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:16.564+0000 7fda16ea9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:16.640 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:04:16.640 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:04:16.640 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:04:16.640 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:04:16.640 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:04:16.640 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:04:16.640 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:04:16.640 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:04:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:04:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:04:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:04:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:04:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:04:16.644 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:04:16.645 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:04:16.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:16.785+0000 7f6e64a6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:16.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:16.796+0000 7f6e64a6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:16.902 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:04:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:04:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:04:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:04:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:04:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:04:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:04:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:04:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:04:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:04:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:04:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:04:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:04:16.905 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:04:16.906 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:04:17.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:17.050+0000 7f8f9bd97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:17.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:17.060+0000 7f8f9bd97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:17.080 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:04:17.080 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:04:17.080 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:04:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:04:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:04:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:04:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:04:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:04:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:04:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:04:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:04:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:04:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:04:17.085 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:04:17.085 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:04:17.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:17.221+0000 7f8e57d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:17.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:17.231+0000 7f8e57d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:04:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:04:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:04:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:04:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:04:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:04:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:04:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:04:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:04:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:04:17.268 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:04:17.268 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:04:17.271 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:04:17.271 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:04:18.748 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:18.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:18.877+0000 7fa62a557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:18.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:18.883+0000 7fa62a557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:21.177 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:21.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:21.320+0000 7f1e36597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:21.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:21.326+0000 7f1e36597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:22.272 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 2] 2024-09-22T03:04:22.273 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:04:22.273 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:04:22.273 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:04:22.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:22.409+0000 7fb8d0b0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:22.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:22.423+0000 7fb8d0b0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:04:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:04:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:04:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:04:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:04:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:04:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:04:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:04:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:04:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:04:22.455 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:04:22.455 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:04:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:04:22.458 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:04:22.458 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:04:22.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:22.600+0000 7ffb0cb57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:22.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:22.611+0000 7ffb0cb57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:22.643 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:04:22.643 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:04:22.643 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:04:22.643 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:04:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:04:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:04:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:04:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:04:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:04:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:04:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:04:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:04:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:04:22.647 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:04:22.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:04:22.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:22.793+0000 7f1886517640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:22.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:22.803+0000 7f1886517640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:22.878 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:04:22.878 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:04:22.878 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:04:22.878 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:04:22.878 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:04:22.878 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:04:22.879 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:04:22.879 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:04:22.879 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:04:22.879 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:04:22.879 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:04:22.879 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:04:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:04:22.881 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:04:22.882 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:04:23.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:23.022+0000 7f520bc6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:23.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:23.032+0000 7f520bc6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:23.095 INFO:teuthology.orchestra.run.smithi081.stderr:no valid command found; 10 closest matches: 2024-09-22T03:04:23.095 INFO:teuthology.orchestra.run.smithi081.stderr:config get 2024-09-22T03:04:23.095 INFO:teuthology.orchestra.run.smithi081.stderr:config help 2024-09-22T03:04:23.095 INFO:teuthology.orchestra.run.smithi081.stderr:config set ... 2024-09-22T03:04:23.095 INFO:teuthology.orchestra.run.smithi081.stderr:config show 2024-09-22T03:04:23.095 INFO:teuthology.orchestra.run.smithi081.stderr:deep_scrub 2024-09-22T03:04:23.095 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_ops 2024-09-22T03:04:23.095 INFO:teuthology.orchestra.run.smithi081.stderr:dump_historic_slow_ops 2024-09-22T03:04:23.095 INFO:teuthology.orchestra.run.smithi081.stderr:dump_metrics [] 2024-09-22T03:04:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:dump_ops_in_flight 2024-09-22T03:04:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:dump_pgstate_history 2024-09-22T03:04:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: invalid command 2024-09-22T03:04:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:04:23.098 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:04:23.515 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:23.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:23.647+0000 7f1710f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:23.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:23.653+0000 7f1710f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:26.009 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:26.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:26.153+0000 7f4e9c423640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:26.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:26.160+0000 7f4e9c423640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:28.099 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 2] 2024-09-22T03:04:28.099 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-22T03:04:28.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:28.212+0000 7f2887b0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:28.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:28.222+0000 7f2887b0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:28.443 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:28.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:28.572+0000 7f4a7e597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:28.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:28.578+0000 7f4a7e597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:29.159 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.0 to 1 (10000) 2024-09-22T03:04:29.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-22T03:04:29.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:29.292+0000 7f94dd46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:29.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:29.307+0000 7f94dd46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:30.159 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.1 to 1 (10000) 2024-09-22T03:04:30.176 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-22T03:04:30.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:30.325+0000 7f39ea997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:30.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:30.337+0000 7f39ea997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:30.729 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:30.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:30.869+0000 7fa175d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:30.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:30.874+0000 7fa175d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:31.167 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.2 to 1 (10000) 2024-09-22T03:04:31.186 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-22T03:04:31.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:31.331+0000 7ff8dc717640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:31.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:31.345+0000 7ff8dc717640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:32.167 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.3 to 1 (10000) 2024-09-22T03:04:32.186 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-22T03:04:32.186 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:32.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:32.331+0000 7fa19b30e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:32.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:32.339+0000 7fa19b30e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:32.564 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:32.564 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:32.582 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-22T03:04:32.354736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":625148491,"num_objects":9535,"num_object_clones":0,"num_object_copies":28605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":52020,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51968,"ondisk_log_size":51968,"up":75,"acting":67,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":311976,"kb_used_data":129188,"kb_used_omap":0,"kb_used_meta":182720,"kb_avail":377175384,"statfs":{"total":386547056640,"available":386227593216,"internally_reserved":0,"allocated":132288512,"data_stored":2090859924,"data_compressed":99263090,"data_compressed_allocated":130654208,"data_compressed_original":2090221568,"omap_allocated":0,"internal_metadata":187105280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-35520512,"num_objects":-542,"num_object_clones":0,"num_object_copies":-1626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":542,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000846"},"pg_stats":[{"pgid":"3.f","version":"46'3245","reported_seq":3378,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-22T03:04:27.260966+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:27.260966+0000","last_peered":"2024-09-22T03:04:27.260966+0000","last_clean":"2024-09-22T03:04:27.260966+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:27.260966+0000","last_undegraded":"2024-09-22T03:04:27.260966+0000","last_fullsized":"2024-09-22T03:04:27.260966+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3245,"log_dups_size":0,"ondisk_log_size":3245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3246,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"47'3246","reported_seq":3366,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:28.532825+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:28.532825+0000","last_peered":"2024-09-22T03:04:28.532825+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:28.532825+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3246,"log_dups_size":0,"ondisk_log_size":3246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37945344,"num_objects":579,"num_object_clones":0,"num_object_copies":1737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3247,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":579}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"47'3283","reported_seq":3402,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.510257+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:28.510257+0000","last_peered":"2024-09-22T03:04:28.510257+0000","last_clean":"2024-09-22T03:04:28.510257+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:28.510257+0000","last_undegraded":"2024-09-22T03:04:28.510257+0000","last_fullsized":"2024-09-22T03:04:28.510257+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3283,"log_dups_size":0,"ondisk_log_size":3283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37421080,"num_objects":572,"num_object_clones":0,"num_object_copies":1716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3284,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"45'3097","reported_seq":3205,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:27.172868+0000","last_change":"2024-09-22T03:04:05.652014+0000","last_active":"2024-09-22T03:04:27.172868+0000","last_peered":"2024-09-22T03:04:27.172868+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:27.172868+0000","last_undegraded":"2024-09-22T03:04:05.225991+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3097,"log_dups_size":0,"ondisk_log_size":3097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39059456,"num_objects":596,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3098,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":596}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"48'3218","reported_seq":3555,"reported_epoch":48,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:29.947304+0000","last_change":"2024-09-22T03:04:27.542949+0000","last_active":"2024-09-22T03:04:29.947304+0000","last_peered":"2024-09-22T03:04:29.947304+0000","last_clean":"2024-09-22T03:02:34.806110+0000","last_became_active":"2024-09-22T03:04:05.614390+0000","last_became_peered":"2024-09-22T03:04:05.614390+0000","last_unstale":"2024-09-22T03:04:29.947304+0000","last_undegraded":"2024-09-22T03:04:29.947304+0000","last_fullsized":"2024-09-22T03:04:05.120934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3218,"log_dups_size":0,"ondisk_log_size":3218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38731776,"num_objects":591,"num_object_clones":0,"num_object_copies":1773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":591}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"46'3172","reported_seq":4259,"reported_epoch":46,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:27.333758+0000","last_change":"2024-09-22T03:04:05.766323+0000","last_active":"2024-09-22T03:04:27.333758+0000","last_peered":"2024-09-22T03:04:27.333758+0000","last_clean":"2024-09-22T03:03:02.467151+0000","last_became_active":"2024-09-22T03:04:05.627124+0000","last_became_peered":"2024-09-22T03:04:05.627124+0000","last_unstale":"2024-09-22T03:04:27.333758+0000","last_undegraded":"2024-09-22T03:04:27.333758+0000","last_fullsized":"2024-09-22T03:04:05.261855+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3172,"log_dups_size":0,"ondisk_log_size":3172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":618,"num_object_clones":0,"num_object_copies":1854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3172,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":488,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":618}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"46'3148","reported_seq":4190,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:27.337011+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:27.337011+0000","last_peered":"2024-09-22T03:04:27.337011+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:27.337011+0000","last_undegraded":"2024-09-22T03:04:27.337011+0000","last_fullsized":"2024-09-22T03:04:27.337011+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38076416,"num_objects":581,"num_object_clones":0,"num_object_copies":1743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3149,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":581}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"47'3292","reported_seq":4431,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.524066+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:28.524066+0000","last_peered":"2024-09-22T03:04:28.524066+0000","last_clean":"2024-09-22T03:04:28.524066+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:28.524066+0000","last_undegraded":"2024-09-22T03:04:28.524066+0000","last_fullsized":"2024-09-22T03:04:28.524066+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":97,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.167812+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:29.167812+0000","last_peered":"2024-09-22T03:04:29.167812+0000","last_clean":"2024-09-22T03:04:29.167812+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:29.167812+0000","last_undegraded":"2024-09-22T03:04:29.167812+0000","last_fullsized":"2024-09-22T03:04:29.167812+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"47'3276","reported_seq":3396,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.587734+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:28.587734+0000","last_peered":"2024-09-22T03:04:28.587734+0000","last_clean":"2024-09-22T03:04:28.587734+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:28.587734+0000","last_undegraded":"2024-09-22T03:04:28.587734+0000","last_fullsized":"2024-09-22T03:04:28.587734+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3276,"log_dups_size":0,"ondisk_log_size":3276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3277,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":120,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-22T03:04:27.167915+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:27.167915+0000","last_peered":"2024-09-22T03:04:27.167915+0000","last_clean":"2024-09-22T03:04:27.167915+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:27.167915+0000","last_undegraded":"2024-09-22T03:04:27.167915+0000","last_fullsized":"2024-09-22T03:04:27.167915+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"46'3320","reported_seq":4487,"reported_epoch":47,"state":"peering","last_fresh":"2024-09-22T03:04:28.181855+0000","last_change":"2024-09-22T03:04:28.176718+0000","last_active":"2024-09-22T03:04:28.176718+0000","last_peered":"2024-09-22T03:04:27.789870+0000","last_clean":"2024-09-22T03:02:34.949467+0000","last_became_active":"2024-09-22T03:04:05.611020+0000","last_became_peered":"2024-09-22T03:04:05.611020+0000","last_unstale":"2024-09-22T03:04:28.181855+0000","last_undegraded":"2024-09-22T03:04:28.181855+0000","last_fullsized":"2024-09-22T03:04:28.181855+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3320,"log_dups_size":0,"ondisk_log_size":3320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":598,"num_object_clones":0,"num_object_copies":1794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":109,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.176075+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:28.176075+0000","last_peered":"2024-09-22T03:04:28.176075+0000","last_clean":"2024-09-22T03:04:28.176075+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:28.176075+0000","last_undegraded":"2024-09-22T03:04:28.176075+0000","last_fullsized":"2024-09-22T03:04:28.176075+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"48'3189","reported_seq":3608,"reported_epoch":48,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:29.954399+0000","last_change":"2024-09-22T03:04:25.522373+0000","last_active":"2024-09-22T03:04:29.954399+0000","last_peered":"2024-09-22T03:04:29.954399+0000","last_clean":"2024-09-22T03:02:34.905097+0000","last_became_active":"2024-09-22T03:04:05.614727+0000","last_became_peered":"2024-09-22T03:04:05.614727+0000","last_unstale":"2024-09-22T03:04:29.954399+0000","last_undegraded":"2024-09-22T03:04:29.954399+0000","last_fullsized":"2024-09-22T03:04:05.116295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38010880,"num_objects":580,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":107,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.169481+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:29.169481+0000","last_peered":"2024-09-22T03:04:29.169481+0000","last_clean":"2024-09-22T03:04:29.169481+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:29.169481+0000","last_undegraded":"2024-09-22T03:04:29.169481+0000","last_fullsized":"2024-09-22T03:04:29.169481+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":243,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-22T03:04:27.202687+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:27.202687+0000","last_peered":"2024-09-22T03:04:27.202687+0000","last_clean":"2024-09-22T03:04:27.202687+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:27.202687+0000","last_undegraded":"2024-09-22T03:04:27.202687+0000","last_fullsized":"2024-09-22T03:04:27.202687+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"48'3483","reported_seq":3605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.793239+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:29.793239+0000","last_peered":"2024-09-22T03:04:29.793239+0000","last_clean":"2024-09-22T03:04:29.793239+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:29.793239+0000","last_undegraded":"2024-09-22T03:04:29.793239+0000","last_fullsized":"2024-09-22T03:04:29.793239+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37617664,"num_objects":574,"num_object_clones":0,"num_object_copies":1722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"47'3292","reported_seq":3390,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:28.434335+0000","last_change":"2024-09-22T03:04:05.634562+0000","last_active":"2024-09-22T03:04:28.434335+0000","last_peered":"2024-09-22T03:04:28.434335+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:28.434335+0000","last_undegraded":"2024-09-22T03:04:05.200849+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41353216,"num_objects":631,"num_object_clones":0,"num_object_copies":1893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":631}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.160920+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:28.160920+0000","last_peered":"2024-09-22T03:04:28.160920+0000","last_clean":"2024-09-22T03:04:28.160920+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:28.160920+0000","last_undegraded":"2024-09-22T03:04:28.160920+0000","last_fullsized":"2024-09-22T03:04:28.160920+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"48'3357","reported_seq":3473,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.979806+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:29.979806+0000","last_peered":"2024-09-22T03:04:29.979806+0000","last_clean":"2024-09-22T03:04:29.979806+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:29.979806+0000","last_undegraded":"2024-09-22T03:04:29.979806+0000","last_fullsized":"2024-09-22T03:04:29.979806+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3357,"log_dups_size":0,"ondisk_log_size":3357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":618,"num_object_clones":0,"num_object_copies":1854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3358,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.171833+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:28.171833+0000","last_peered":"2024-09-22T03:04:28.171833+0000","last_clean":"2024-09-22T03:04:28.171833+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:28.171833+0000","last_undegraded":"2024-09-22T03:04:28.171833+0000","last_fullsized":"2024-09-22T03:04:28.171833+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"46'3173","reported_seq":3307,"reported_epoch":46,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:27.334129+0000","last_change":"2024-09-22T03:04:26.435043+0000","last_active":"2024-09-22T03:04:27.334129+0000","last_peered":"2024-09-22T03:04:27.334129+0000","last_clean":"2024-09-22T03:02:34.870768+0000","last_became_active":"2024-09-22T03:04:05.621831+0000","last_became_peered":"2024-09-22T03:04:05.621831+0000","last_unstale":"2024-09-22T03:04:27.334129+0000","last_undegraded":"2024-09-22T03:04:27.334129+0000","last_fullsized":"2024-09-22T03:04:05.135956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40173568,"num_objects":613,"num_object_clones":0,"num_object_copies":1839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3173,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":613}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":84,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.162130+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:28.162130+0000","last_peered":"2024-09-22T03:04:28.162130+0000","last_clean":"2024-09-22T03:04:28.162130+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:28.162130+0000","last_undegraded":"2024-09-22T03:04:28.162130+0000","last_fullsized":"2024-09-22T03:04:28.162130+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"47'3077","reported_seq":3184,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:28.575154+0000","last_change":"2024-09-22T03:04:05.673295+0000","last_active":"2024-09-22T03:04:28.575154+0000","last_peered":"2024-09-22T03:04:28.575154+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:28.575154+0000","last_undegraded":"2024-09-22T03:04:05.671962+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3077,"log_dups_size":0,"ondisk_log_size":3077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37093376,"num_objects":566,"num_object_clones":0,"num_object_copies":1698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":566}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.167452+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:28.167452+0000","last_peered":"2024-09-22T03:04:28.167452+0000","last_clean":"2024-09-22T03:04:28.167452+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:28.167452+0000","last_undegraded":"2024-09-22T03:04:28.167452+0000","last_fullsized":"2024-09-22T03:04:28.167452+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":624558104,"num_objects":9531,"num_object_clones":0,"num_object_copies":28593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":51881,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":129282048,"data_stored":2068316232,"data_compressed":98214717,"data_compressed_allocated":129269760,"data_compressed_original":2068316160,"omap_allocated":0,"internal_metadata":0},"log_size":51868,"ondisk_log_size":51868,"up":48,"acting":40,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70332,"kb_used_data":33708,"kb_used_omap":0,"kb_used_meta":36608,"kb_avail":94301508,"statfs":{"total":96636764160,"available":96564744192,"internally_reserved":0,"allocated":34516992,"data_stored":545935103,"data_compressed":25916736,"data_compressed_allocated":34111488,"data_compressed_original":545783808,"omap_allocated":0,"internal_metadata":37486592},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86780,"kb_used_data":38888,"kb_used_omap":0,"kb_used_meta":47872,"kb_avail":94285060,"statfs":{"total":96636764160,"available":96547901440,"internally_reserved":0,"allocated":39821312,"data_stored":630604643,"data_compressed":29939390,"data_compressed_allocated":39407616,"data_compressed_original":630439936,"omap_allocated":0,"internal_metadata":49020928},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673304,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94536,"kb_used_data":27448,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94277304,"statfs":{"total":96636764160,"available":96539959296,"internally_reserved":0,"allocated":28106752,"data_stored":443426743,"data_compressed":21051518,"data_compressed_allocated":27709440,"data_compressed_original":443269120,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888198,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60328,"kb_used_data":29144,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94311512,"statfs":{"total":96636764160,"available":96574988288,"internally_reserved":0,"allocated":29843456,"data_stored":470893435,"data_compressed":22355446,"data_compressed_allocated":29425664,"data_compressed_original":470728704,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27381760,"data_stored":438042648,"data_compressed":20800607,"data_compressed_allocated":27377664,"data_compressed_original":438042624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29179904,"data_stored":466812952,"data_compressed":22166775,"data_compressed_allocated":29175808,"data_compressed_original":466812928,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39178240,"data_stored":626851840,"data_compressed":29766279,"data_compressed_allocated":39178240,"data_compressed_original":626851840,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33542144,"data_stored":536608792,"data_compressed":25481056,"data_compressed_allocated":33538048,"data_compressed_original":536608768,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:32.583 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:32.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:32.707+0000 7f988422c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:32.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:32.719+0000 7f988422c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:32.983 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:32.984 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:33.002 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-22T03:04:32.354736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":625148491,"num_objects":9535,"num_object_clones":0,"num_object_copies":28605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":52020,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51968,"ondisk_log_size":51968,"up":75,"acting":67,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":311976,"kb_used_data":129188,"kb_used_omap":0,"kb_used_meta":182720,"kb_avail":377175384,"statfs":{"total":386547056640,"available":386227593216,"internally_reserved":0,"allocated":132288512,"data_stored":2090859924,"data_compressed":99263090,"data_compressed_allocated":130654208,"data_compressed_original":2090221568,"omap_allocated":0,"internal_metadata":187105280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-35520512,"num_objects":-542,"num_object_clones":0,"num_object_copies":-1626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":542,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000846"},"pg_stats":[{"pgid":"3.f","version":"46'3245","reported_seq":3378,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-22T03:04:27.260966+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:27.260966+0000","last_peered":"2024-09-22T03:04:27.260966+0000","last_clean":"2024-09-22T03:04:27.260966+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:27.260966+0000","last_undegraded":"2024-09-22T03:04:27.260966+0000","last_fullsized":"2024-09-22T03:04:27.260966+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3245,"log_dups_size":0,"ondisk_log_size":3245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3246,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"47'3246","reported_seq":3366,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:28.532825+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:28.532825+0000","last_peered":"2024-09-22T03:04:28.532825+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:28.532825+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3246,"log_dups_size":0,"ondisk_log_size":3246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37945344,"num_objects":579,"num_object_clones":0,"num_object_copies":1737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3247,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":579}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"47'3283","reported_seq":3402,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.510257+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:28.510257+0000","last_peered":"2024-09-22T03:04:28.510257+0000","last_clean":"2024-09-22T03:04:28.510257+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:28.510257+0000","last_undegraded":"2024-09-22T03:04:28.510257+0000","last_fullsized":"2024-09-22T03:04:28.510257+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3283,"log_dups_size":0,"ondisk_log_size":3283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37421080,"num_objects":572,"num_object_clones":0,"num_object_copies":1716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3284,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"45'3097","reported_seq":3205,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:27.172868+0000","last_change":"2024-09-22T03:04:05.652014+0000","last_active":"2024-09-22T03:04:27.172868+0000","last_peered":"2024-09-22T03:04:27.172868+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:27.172868+0000","last_undegraded":"2024-09-22T03:04:05.225991+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3097,"log_dups_size":0,"ondisk_log_size":3097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39059456,"num_objects":596,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3098,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":596}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"48'3218","reported_seq":3555,"reported_epoch":48,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:29.947304+0000","last_change":"2024-09-22T03:04:27.542949+0000","last_active":"2024-09-22T03:04:29.947304+0000","last_peered":"2024-09-22T03:04:29.947304+0000","last_clean":"2024-09-22T03:02:34.806110+0000","last_became_active":"2024-09-22T03:04:05.614390+0000","last_became_peered":"2024-09-22T03:04:05.614390+0000","last_unstale":"2024-09-22T03:04:29.947304+0000","last_undegraded":"2024-09-22T03:04:29.947304+0000","last_fullsized":"2024-09-22T03:04:05.120934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3218,"log_dups_size":0,"ondisk_log_size":3218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38731776,"num_objects":591,"num_object_clones":0,"num_object_copies":1773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":591}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"46'3172","reported_seq":4259,"reported_epoch":46,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:27.333758+0000","last_change":"2024-09-22T03:04:05.766323+0000","last_active":"2024-09-22T03:04:27.333758+0000","last_peered":"2024-09-22T03:04:27.333758+0000","last_clean":"2024-09-22T03:03:02.467151+0000","last_became_active":"2024-09-22T03:04:05.627124+0000","last_became_peered":"2024-09-22T03:04:05.627124+0000","last_unstale":"2024-09-22T03:04:27.333758+0000","last_undegraded":"2024-09-22T03:04:27.333758+0000","last_fullsized":"2024-09-22T03:04:05.261855+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3172,"log_dups_size":0,"ondisk_log_size":3172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":618,"num_object_clones":0,"num_object_copies":1854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3172,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":488,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":618}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"46'3148","reported_seq":4190,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:27.337011+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:27.337011+0000","last_peered":"2024-09-22T03:04:27.337011+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:27.337011+0000","last_undegraded":"2024-09-22T03:04:27.337011+0000","last_fullsized":"2024-09-22T03:04:27.337011+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38076416,"num_objects":581,"num_object_clones":0,"num_object_copies":1743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3149,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":581}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"47'3292","reported_seq":4431,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.524066+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:28.524066+0000","last_peered":"2024-09-22T03:04:28.524066+0000","last_clean":"2024-09-22T03:04:28.524066+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:28.524066+0000","last_undegraded":"2024-09-22T03:04:28.524066+0000","last_fullsized":"2024-09-22T03:04:28.524066+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":97,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.167812+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:29.167812+0000","last_peered":"2024-09-22T03:04:29.167812+0000","last_clean":"2024-09-22T03:04:29.167812+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:29.167812+0000","last_undegraded":"2024-09-22T03:04:29.167812+0000","last_fullsized":"2024-09-22T03:04:29.167812+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"47'3276","reported_seq":3396,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.587734+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:28.587734+0000","last_peered":"2024-09-22T03:04:28.587734+0000","last_clean":"2024-09-22T03:04:28.587734+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:28.587734+0000","last_undegraded":"2024-09-22T03:04:28.587734+0000","last_fullsized":"2024-09-22T03:04:28.587734+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3276,"log_dups_size":0,"ondisk_log_size":3276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3277,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":120,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-22T03:04:27.167915+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:27.167915+0000","last_peered":"2024-09-22T03:04:27.167915+0000","last_clean":"2024-09-22T03:04:27.167915+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:27.167915+0000","last_undegraded":"2024-09-22T03:04:27.167915+0000","last_fullsized":"2024-09-22T03:04:27.167915+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"46'3320","reported_seq":4487,"reported_epoch":47,"state":"peering","last_fresh":"2024-09-22T03:04:28.181855+0000","last_change":"2024-09-22T03:04:28.176718+0000","last_active":"2024-09-22T03:04:28.176718+0000","last_peered":"2024-09-22T03:04:27.789870+0000","last_clean":"2024-09-22T03:02:34.949467+0000","last_became_active":"2024-09-22T03:04:05.611020+0000","last_became_peered":"2024-09-22T03:04:05.611020+0000","last_unstale":"2024-09-22T03:04:28.181855+0000","last_undegraded":"2024-09-22T03:04:28.181855+0000","last_fullsized":"2024-09-22T03:04:28.181855+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3320,"log_dups_size":0,"ondisk_log_size":3320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":598,"num_object_clones":0,"num_object_copies":1794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":109,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.176075+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:28.176075+0000","last_peered":"2024-09-22T03:04:28.176075+0000","last_clean":"2024-09-22T03:04:28.176075+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:28.176075+0000","last_undegraded":"2024-09-22T03:04:28.176075+0000","last_fullsized":"2024-09-22T03:04:28.176075+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"48'3189","reported_seq":3608,"reported_epoch":48,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:29.954399+0000","last_change":"2024-09-22T03:04:25.522373+0000","last_active":"2024-09-22T03:04:29.954399+0000","last_peered":"2024-09-22T03:04:29.954399+0000","last_clean":"2024-09-22T03:02:34.905097+0000","last_became_active":"2024-09-22T03:04:05.614727+0000","last_became_peered":"2024-09-22T03:04:05.614727+0000","last_unstale":"2024-09-22T03:04:29.954399+0000","last_undegraded":"2024-09-22T03:04:29.954399+0000","last_fullsized":"2024-09-22T03:04:05.116295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38010880,"num_objects":580,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":107,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.169481+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:29.169481+0000","last_peered":"2024-09-22T03:04:29.169481+0000","last_clean":"2024-09-22T03:04:29.169481+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:29.169481+0000","last_undegraded":"2024-09-22T03:04:29.169481+0000","last_fullsized":"2024-09-22T03:04:29.169481+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":243,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-22T03:04:27.202687+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:27.202687+0000","last_peered":"2024-09-22T03:04:27.202687+0000","last_clean":"2024-09-22T03:04:27.202687+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:27.202687+0000","last_undegraded":"2024-09-22T03:04:27.202687+0000","last_fullsized":"2024-09-22T03:04:27.202687+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"48'3483","reported_seq":3605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.793239+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:29.793239+0000","last_peered":"2024-09-22T03:04:29.793239+0000","last_clean":"2024-09-22T03:04:29.793239+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:29.793239+0000","last_undegraded":"2024-09-22T03:04:29.793239+0000","last_fullsized":"2024-09-22T03:04:29.793239+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37617664,"num_objects":574,"num_object_clones":0,"num_object_copies":1722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"47'3292","reported_seq":3390,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:28.434335+0000","last_change":"2024-09-22T03:04:05.634562+0000","last_active":"2024-09-22T03:04:28.434335+0000","last_peered":"2024-09-22T03:04:28.434335+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:28.434335+0000","last_undegraded":"2024-09-22T03:04:05.200849+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41353216,"num_objects":631,"num_object_clones":0,"num_object_copies":1893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":631}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.160920+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:28.160920+0000","last_peered":"2024-09-22T03:04:28.160920+0000","last_clean":"2024-09-22T03:04:28.160920+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:28.160920+0000","last_undegraded":"2024-09-22T03:04:28.160920+0000","last_fullsized":"2024-09-22T03:04:28.160920+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"48'3357","reported_seq":3473,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.979806+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:29.979806+0000","last_peered":"2024-09-22T03:04:29.979806+0000","last_clean":"2024-09-22T03:04:29.979806+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:29.979806+0000","last_undegraded":"2024-09-22T03:04:29.979806+0000","last_fullsized":"2024-09-22T03:04:29.979806+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3357,"log_dups_size":0,"ondisk_log_size":3357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":618,"num_object_clones":0,"num_object_copies":1854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3358,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.171833+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:28.171833+0000","last_peered":"2024-09-22T03:04:28.171833+0000","last_clean":"2024-09-22T03:04:28.171833+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:28.171833+0000","last_undegraded":"2024-09-22T03:04:28.171833+0000","last_fullsized":"2024-09-22T03:04:28.171833+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"46'3173","reported_seq":3307,"reported_epoch":46,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:27.334129+0000","last_change":"2024-09-22T03:04:26.435043+0000","last_active":"2024-09-22T03:04:27.334129+0000","last_peered":"2024-09-22T03:04:27.334129+0000","last_clean":"2024-09-22T03:02:34.870768+0000","last_became_active":"2024-09-22T03:04:05.621831+0000","last_became_peered":"2024-09-22T03:04:05.621831+0000","last_unstale":"2024-09-22T03:04:27.334129+0000","last_undegraded":"2024-09-22T03:04:27.334129+0000","last_fullsized":"2024-09-22T03:04:05.135956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40173568,"num_objects":613,"num_object_clones":0,"num_object_copies":1839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3173,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":613}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":84,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.162130+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:28.162130+0000","last_peered":"2024-09-22T03:04:28.162130+0000","last_clean":"2024-09-22T03:04:28.162130+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:28.162130+0000","last_undegraded":"2024-09-22T03:04:28.162130+0000","last_fullsized":"2024-09-22T03:04:28.162130+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"47'3077","reported_seq":3184,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:28.575154+0000","last_change":"2024-09-22T03:04:05.673295+0000","last_active":"2024-09-22T03:04:28.575154+0000","last_peered":"2024-09-22T03:04:28.575154+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:28.575154+0000","last_undegraded":"2024-09-22T03:04:05.671962+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3077,"log_dups_size":0,"ondisk_log_size":3077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37093376,"num_objects":566,"num_object_clones":0,"num_object_copies":1698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":566}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.167452+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:28.167452+0000","last_peered":"2024-09-22T03:04:28.167452+0000","last_clean":"2024-09-22T03:04:28.167452+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:28.167452+0000","last_undegraded":"2024-09-22T03:04:28.167452+0000","last_fullsized":"2024-09-22T03:04:28.167452+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":624558104,"num_objects":9531,"num_object_clones":0,"num_object_copies":28593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":51881,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":129282048,"data_stored":2068316232,"data_compressed":98214717,"data_compressed_allocated":129269760,"data_compressed_original":2068316160,"omap_allocated":0,"internal_metadata":0},"log_size":51868,"ondisk_log_size":51868,"up":48,"acting":40,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70332,"kb_used_data":33708,"kb_used_omap":0,"kb_used_meta":36608,"kb_avail":94301508,"statfs":{"total":96636764160,"available":96564744192,"internally_reserved":0,"allocated":34516992,"data_stored":545935103,"data_compressed":25916736,"data_compressed_allocated":34111488,"data_compressed_original":545783808,"omap_allocated":0,"internal_metadata":37486592},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86780,"kb_used_data":38888,"kb_used_omap":0,"kb_used_meta":47872,"kb_avail":94285060,"statfs":{"total":96636764160,"available":96547901440,"internally_reserved":0,"allocated":39821312,"data_stored":630604643,"data_compressed":29939390,"data_compressed_allocated":39407616,"data_compressed_original":630439936,"omap_allocated":0,"internal_metadata":49020928},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673304,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94536,"kb_used_data":27448,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94277304,"statfs":{"total":96636764160,"available":96539959296,"internally_reserved":0,"allocated":28106752,"data_stored":443426743,"data_compressed":21051518,"data_compressed_allocated":27709440,"data_compressed_original":443269120,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888198,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60328,"kb_used_data":29144,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94311512,"statfs":{"total":96636764160,"available":96574988288,"internally_reserved":0,"allocated":29843456,"data_stored":470893435,"data_compressed":22355446,"data_compressed_allocated":29425664,"data_compressed_original":470728704,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27381760,"data_stored":438042648,"data_compressed":20800607,"data_compressed_allocated":27377664,"data_compressed_original":438042624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29179904,"data_stored":466812952,"data_compressed":22166775,"data_compressed_allocated":29175808,"data_compressed_original":466812928,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39178240,"data_stored":626851840,"data_compressed":29766279,"data_compressed_allocated":39178240,"data_compressed_original":626851840,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33542144,"data_stored":536608792,"data_compressed":25481056,"data_compressed_allocated":33538048,"data_compressed_original":536608768,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:33.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:04:33.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:33.126+0000 7f829ad17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:33.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:33.136+0000 7f829ad17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:33.299 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:33.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:33.444+0000 7f84edb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:33.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:33.452+0000 7f84edb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:33.565 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:33.565 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1921/28605 objects degraded (6.716%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":359,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974243,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+remapped","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovery_wait","count":1},{"state_name":"peering","count":1}],"num_pgs":25,"num_pools":3,"num_objects":9535,"data_bytes":625148491,"bytes_used":319463424,"bytes_avail":386227593216,"bytes_total":386547056640,"inactive_pgs_ratio":0.039999999105930328,"degraded_objects":1921,"degraded_total":28605,"degraded_ratio":0.067156091592378953,"recovering_objects_per_sec":81,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":328,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":135},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:04:33.586 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:04:33.586 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:33.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:33.735+0000 7fe7ca397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:33.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:33.745+0000 7fe7ca397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:34.034 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:34.035 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:34.052 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-22T03:04:32.354736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":625148491,"num_objects":9535,"num_object_clones":0,"num_object_copies":28605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":52020,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51968,"ondisk_log_size":51968,"up":75,"acting":67,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":311976,"kb_used_data":129188,"kb_used_omap":0,"kb_used_meta":182720,"kb_avail":377175384,"statfs":{"total":386547056640,"available":386227593216,"internally_reserved":0,"allocated":132288512,"data_stored":2090859924,"data_compressed":99263090,"data_compressed_allocated":130654208,"data_compressed_original":2090221568,"omap_allocated":0,"internal_metadata":187105280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-35520512,"num_objects":-542,"num_object_clones":0,"num_object_copies":-1626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":542,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000846"},"pg_stats":[{"pgid":"3.f","version":"46'3245","reported_seq":3378,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-22T03:04:27.260966+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:27.260966+0000","last_peered":"2024-09-22T03:04:27.260966+0000","last_clean":"2024-09-22T03:04:27.260966+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:27.260966+0000","last_undegraded":"2024-09-22T03:04:27.260966+0000","last_fullsized":"2024-09-22T03:04:27.260966+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3245,"log_dups_size":0,"ondisk_log_size":3245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3246,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"47'3246","reported_seq":3366,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:28.532825+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:28.532825+0000","last_peered":"2024-09-22T03:04:28.532825+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:28.532825+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3246,"log_dups_size":0,"ondisk_log_size":3246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37945344,"num_objects":579,"num_object_clones":0,"num_object_copies":1737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3247,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":579}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"47'3283","reported_seq":3402,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.510257+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:28.510257+0000","last_peered":"2024-09-22T03:04:28.510257+0000","last_clean":"2024-09-22T03:04:28.510257+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:28.510257+0000","last_undegraded":"2024-09-22T03:04:28.510257+0000","last_fullsized":"2024-09-22T03:04:28.510257+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3283,"log_dups_size":0,"ondisk_log_size":3283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37421080,"num_objects":572,"num_object_clones":0,"num_object_copies":1716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3284,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"45'3097","reported_seq":3205,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:27.172868+0000","last_change":"2024-09-22T03:04:05.652014+0000","last_active":"2024-09-22T03:04:27.172868+0000","last_peered":"2024-09-22T03:04:27.172868+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:27.172868+0000","last_undegraded":"2024-09-22T03:04:05.225991+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3097,"log_dups_size":0,"ondisk_log_size":3097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39059456,"num_objects":596,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3098,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":596}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"48'3218","reported_seq":3555,"reported_epoch":48,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:29.947304+0000","last_change":"2024-09-22T03:04:27.542949+0000","last_active":"2024-09-22T03:04:29.947304+0000","last_peered":"2024-09-22T03:04:29.947304+0000","last_clean":"2024-09-22T03:02:34.806110+0000","last_became_active":"2024-09-22T03:04:05.614390+0000","last_became_peered":"2024-09-22T03:04:05.614390+0000","last_unstale":"2024-09-22T03:04:29.947304+0000","last_undegraded":"2024-09-22T03:04:29.947304+0000","last_fullsized":"2024-09-22T03:04:05.120934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3218,"log_dups_size":0,"ondisk_log_size":3218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38731776,"num_objects":591,"num_object_clones":0,"num_object_copies":1773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":591}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"46'3172","reported_seq":4259,"reported_epoch":46,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:27.333758+0000","last_change":"2024-09-22T03:04:05.766323+0000","last_active":"2024-09-22T03:04:27.333758+0000","last_peered":"2024-09-22T03:04:27.333758+0000","last_clean":"2024-09-22T03:03:02.467151+0000","last_became_active":"2024-09-22T03:04:05.627124+0000","last_became_peered":"2024-09-22T03:04:05.627124+0000","last_unstale":"2024-09-22T03:04:27.333758+0000","last_undegraded":"2024-09-22T03:04:27.333758+0000","last_fullsized":"2024-09-22T03:04:05.261855+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3172,"log_dups_size":0,"ondisk_log_size":3172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":618,"num_object_clones":0,"num_object_copies":1854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3172,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":488,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":618}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"46'3148","reported_seq":4190,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:27.337011+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:27.337011+0000","last_peered":"2024-09-22T03:04:27.337011+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:27.337011+0000","last_undegraded":"2024-09-22T03:04:27.337011+0000","last_fullsized":"2024-09-22T03:04:27.337011+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38076416,"num_objects":581,"num_object_clones":0,"num_object_copies":1743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3149,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":581}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"47'3292","reported_seq":4431,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.524066+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:28.524066+0000","last_peered":"2024-09-22T03:04:28.524066+0000","last_clean":"2024-09-22T03:04:28.524066+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:28.524066+0000","last_undegraded":"2024-09-22T03:04:28.524066+0000","last_fullsized":"2024-09-22T03:04:28.524066+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":97,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.167812+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:29.167812+0000","last_peered":"2024-09-22T03:04:29.167812+0000","last_clean":"2024-09-22T03:04:29.167812+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:29.167812+0000","last_undegraded":"2024-09-22T03:04:29.167812+0000","last_fullsized":"2024-09-22T03:04:29.167812+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"47'3276","reported_seq":3396,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.587734+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:28.587734+0000","last_peered":"2024-09-22T03:04:28.587734+0000","last_clean":"2024-09-22T03:04:28.587734+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:28.587734+0000","last_undegraded":"2024-09-22T03:04:28.587734+0000","last_fullsized":"2024-09-22T03:04:28.587734+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3276,"log_dups_size":0,"ondisk_log_size":3276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3277,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":120,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-22T03:04:27.167915+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:27.167915+0000","last_peered":"2024-09-22T03:04:27.167915+0000","last_clean":"2024-09-22T03:04:27.167915+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:27.167915+0000","last_undegraded":"2024-09-22T03:04:27.167915+0000","last_fullsized":"2024-09-22T03:04:27.167915+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"46'3320","reported_seq":4487,"reported_epoch":47,"state":"peering","last_fresh":"2024-09-22T03:04:28.181855+0000","last_change":"2024-09-22T03:04:28.176718+0000","last_active":"2024-09-22T03:04:28.176718+0000","last_peered":"2024-09-22T03:04:27.789870+0000","last_clean":"2024-09-22T03:02:34.949467+0000","last_became_active":"2024-09-22T03:04:05.611020+0000","last_became_peered":"2024-09-22T03:04:05.611020+0000","last_unstale":"2024-09-22T03:04:28.181855+0000","last_undegraded":"2024-09-22T03:04:28.181855+0000","last_fullsized":"2024-09-22T03:04:28.181855+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3320,"log_dups_size":0,"ondisk_log_size":3320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":598,"num_object_clones":0,"num_object_copies":1794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":109,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.176075+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:28.176075+0000","last_peered":"2024-09-22T03:04:28.176075+0000","last_clean":"2024-09-22T03:04:28.176075+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:28.176075+0000","last_undegraded":"2024-09-22T03:04:28.176075+0000","last_fullsized":"2024-09-22T03:04:28.176075+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"48'3189","reported_seq":3608,"reported_epoch":48,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:29.954399+0000","last_change":"2024-09-22T03:04:25.522373+0000","last_active":"2024-09-22T03:04:29.954399+0000","last_peered":"2024-09-22T03:04:29.954399+0000","last_clean":"2024-09-22T03:02:34.905097+0000","last_became_active":"2024-09-22T03:04:05.614727+0000","last_became_peered":"2024-09-22T03:04:05.614727+0000","last_unstale":"2024-09-22T03:04:29.954399+0000","last_undegraded":"2024-09-22T03:04:29.954399+0000","last_fullsized":"2024-09-22T03:04:05.116295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38010880,"num_objects":580,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":107,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.169481+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:29.169481+0000","last_peered":"2024-09-22T03:04:29.169481+0000","last_clean":"2024-09-22T03:04:29.169481+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:29.169481+0000","last_undegraded":"2024-09-22T03:04:29.169481+0000","last_fullsized":"2024-09-22T03:04:29.169481+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":243,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-22T03:04:27.202687+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:27.202687+0000","last_peered":"2024-09-22T03:04:27.202687+0000","last_clean":"2024-09-22T03:04:27.202687+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:27.202687+0000","last_undegraded":"2024-09-22T03:04:27.202687+0000","last_fullsized":"2024-09-22T03:04:27.202687+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"48'3483","reported_seq":3605,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.793239+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:29.793239+0000","last_peered":"2024-09-22T03:04:29.793239+0000","last_clean":"2024-09-22T03:04:29.793239+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:29.793239+0000","last_undegraded":"2024-09-22T03:04:29.793239+0000","last_fullsized":"2024-09-22T03:04:29.793239+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37617664,"num_objects":574,"num_object_clones":0,"num_object_copies":1722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"47'3292","reported_seq":3390,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:28.434335+0000","last_change":"2024-09-22T03:04:05.634562+0000","last_active":"2024-09-22T03:04:28.434335+0000","last_peered":"2024-09-22T03:04:28.434335+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:28.434335+0000","last_undegraded":"2024-09-22T03:04:05.200849+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41353216,"num_objects":631,"num_object_clones":0,"num_object_copies":1893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":631}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.160920+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:28.160920+0000","last_peered":"2024-09-22T03:04:28.160920+0000","last_clean":"2024-09-22T03:04:28.160920+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:28.160920+0000","last_undegraded":"2024-09-22T03:04:28.160920+0000","last_fullsized":"2024-09-22T03:04:28.160920+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"48'3357","reported_seq":3473,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-22T03:04:29.979806+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:29.979806+0000","last_peered":"2024-09-22T03:04:29.979806+0000","last_clean":"2024-09-22T03:04:29.979806+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:29.979806+0000","last_undegraded":"2024-09-22T03:04:29.979806+0000","last_fullsized":"2024-09-22T03:04:29.979806+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3357,"log_dups_size":0,"ondisk_log_size":3357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":618,"num_object_clones":0,"num_object_copies":1854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3358,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.171833+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:28.171833+0000","last_peered":"2024-09-22T03:04:28.171833+0000","last_clean":"2024-09-22T03:04:28.171833+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:28.171833+0000","last_undegraded":"2024-09-22T03:04:28.171833+0000","last_fullsized":"2024-09-22T03:04:28.171833+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"46'3173","reported_seq":3307,"reported_epoch":46,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:27.334129+0000","last_change":"2024-09-22T03:04:26.435043+0000","last_active":"2024-09-22T03:04:27.334129+0000","last_peered":"2024-09-22T03:04:27.334129+0000","last_clean":"2024-09-22T03:02:34.870768+0000","last_became_active":"2024-09-22T03:04:05.621831+0000","last_became_peered":"2024-09-22T03:04:05.621831+0000","last_unstale":"2024-09-22T03:04:27.334129+0000","last_undegraded":"2024-09-22T03:04:27.334129+0000","last_fullsized":"2024-09-22T03:04:05.135956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40173568,"num_objects":613,"num_object_clones":0,"num_object_copies":1839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3173,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":613}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":84,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.162130+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:28.162130+0000","last_peered":"2024-09-22T03:04:28.162130+0000","last_clean":"2024-09-22T03:04:28.162130+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:28.162130+0000","last_undegraded":"2024-09-22T03:04:28.162130+0000","last_fullsized":"2024-09-22T03:04:28.162130+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"47'3077","reported_seq":3184,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:28.575154+0000","last_change":"2024-09-22T03:04:05.673295+0000","last_active":"2024-09-22T03:04:28.575154+0000","last_peered":"2024-09-22T03:04:28.575154+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:28.575154+0000","last_undegraded":"2024-09-22T03:04:05.671962+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3077,"log_dups_size":0,"ondisk_log_size":3077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37093376,"num_objects":566,"num_object_clones":0,"num_object_copies":1698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":566}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-22T03:04:28.167452+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:28.167452+0000","last_peered":"2024-09-22T03:04:28.167452+0000","last_clean":"2024-09-22T03:04:28.167452+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:28.167452+0000","last_undegraded":"2024-09-22T03:04:28.167452+0000","last_fullsized":"2024-09-22T03:04:28.167452+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":624558104,"num_objects":9531,"num_object_clones":0,"num_object_copies":28593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":51881,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2312,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":129282048,"data_stored":2068316232,"data_compressed":98214717,"data_compressed_allocated":129269760,"data_compressed_original":2068316160,"omap_allocated":0,"internal_metadata":0},"log_size":51868,"ondisk_log_size":51868,"up":48,"acting":40,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70332,"kb_used_data":33708,"kb_used_omap":0,"kb_used_meta":36608,"kb_avail":94301508,"statfs":{"total":96636764160,"available":96564744192,"internally_reserved":0,"allocated":34516992,"data_stored":545935103,"data_compressed":25916736,"data_compressed_allocated":34111488,"data_compressed_original":545783808,"omap_allocated":0,"internal_metadata":37486592},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86780,"kb_used_data":38888,"kb_used_omap":0,"kb_used_meta":47872,"kb_avail":94285060,"statfs":{"total":96636764160,"available":96547901440,"internally_reserved":0,"allocated":39821312,"data_stored":630604643,"data_compressed":29939390,"data_compressed_allocated":39407616,"data_compressed_original":630439936,"omap_allocated":0,"internal_metadata":49020928},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673304,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94536,"kb_used_data":27448,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94277304,"statfs":{"total":96636764160,"available":96539959296,"internally_reserved":0,"allocated":28106752,"data_stored":443426743,"data_compressed":21051518,"data_compressed_allocated":27709440,"data_compressed_original":443269120,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888198,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60328,"kb_used_data":29144,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94311512,"statfs":{"total":96636764160,"available":96574988288,"internally_reserved":0,"allocated":29843456,"data_stored":470893435,"data_compressed":22355446,"data_compressed_allocated":29425664,"data_compressed_original":470728704,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27381760,"data_stored":438042648,"data_compressed":20800607,"data_compressed_allocated":27377664,"data_compressed_original":438042624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29179904,"data_stored":466812952,"data_compressed":22166775,"data_compressed_allocated":29175808,"data_compressed_original":466812928,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39178240,"data_stored":626851840,"data_compressed":29766279,"data_compressed_allocated":39178240,"data_compressed_original":626851840,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33542144,"data_stored":536608792,"data_compressed":25481056,"data_compressed_allocated":33538048,"data_compressed_original":536608768,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:35.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:35.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:35.841+0000 7fd9e6317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:35.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:35.848+0000 7fd9e6317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:37.055 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:37.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:37.197+0000 7fe3af997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:37.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:37.208+0000 7fe3af997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:37.439 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:37.440 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:37.454 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-22T03:04:36.355510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562365003,"num_objects":8577,"num_object_clones":0,"num_object_copies":25731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":52978,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2717,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52926,"ondisk_log_size":52926,"up":75,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":299332,"kb_used_data":116536,"kb_used_omap":0,"kb_used_meta":182720,"kb_avail":377188028,"statfs":{"total":386547056640,"available":386240540672,"internally_reserved":0,"allocated":119332864,"data_stored":1881528205,"data_compressed":89320250,"data_compressed_allocated":117567488,"data_compressed_original":1880834048,"omap_allocated":0,"internal_metadata":187105280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62783488,"num_objects":-958,"num_object_clones":0,"num_object_copies":-2874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":958,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.196297"},"pg_stats":[{"pgid":"3.f","version":"50'3297","reported_seq":3444,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.204727+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:32.204727+0000","last_peered":"2024-09-22T03:04:32.204727+0000","last_clean":"2024-09-22T03:04:32.204727+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:32.204727+0000","last_undegraded":"2024-09-22T03:04:32.204727+0000","last_fullsized":"2024-09-22T03:04:32.204727+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38338560,"num_objects":585,"num_object_clones":0,"num_object_copies":1755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3297,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"51'3306","reported_seq":3434,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:33.460805+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:33.460805+0000","last_peered":"2024-09-22T03:04:33.460805+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:33.460805+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":519,"num_object_clones":0,"num_object_copies":1557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3307,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":519}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"51'3339","reported_seq":3470,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:33.695222+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:33.695222+0000","last_peered":"2024-09-22T03:04:33.695222+0000","last_clean":"2024-09-22T03:04:33.695222+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:33.695222+0000","last_undegraded":"2024-09-22T03:04:33.695222+0000","last_fullsized":"2024-09-22T03:04:33.695222+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3339,"log_dups_size":0,"ondisk_log_size":3339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751064,"num_objects":516,"num_object_clones":0,"num_object_copies":1548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3340,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"50'3158","reported_seq":3275,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:32.205253+0000","last_change":"2024-09-22T03:04:05.652014+0000","last_active":"2024-09-22T03:04:32.205253+0000","last_peered":"2024-09-22T03:04:32.205253+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:32.205253+0000","last_undegraded":"2024-09-22T03:04:05.225991+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3158,"log_dups_size":0,"ondisk_log_size":3158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":536,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3158,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":536}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"51'3276","reported_seq":3866,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:34.992176+0000","last_change":"2024-09-22T03:04:27.542949+0000","last_active":"2024-09-22T03:04:34.992176+0000","last_peered":"2024-09-22T03:04:34.992176+0000","last_clean":"2024-09-22T03:02:34.806110+0000","last_became_active":"2024-09-22T03:04:05.614390+0000","last_became_peered":"2024-09-22T03:04:05.614390+0000","last_unstale":"2024-09-22T03:04:34.992176+0000","last_undegraded":"2024-09-22T03:04:34.992176+0000","last_fullsized":"2024-09-22T03:04:05.120934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3276,"log_dups_size":0,"ondisk_log_size":3276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34996224,"num_objects":534,"num_object_clones":0,"num_object_copies":1602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":534}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"51'3224","reported_seq":4339,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.273770+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:32.273770+0000","last_peered":"2024-09-22T03:04:32.273770+0000","last_clean":"2024-09-22T03:04:32.273770+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:32.273770+0000","last_undegraded":"2024-09-22T03:04:32.273770+0000","last_fullsized":"2024-09-22T03:04:32.273770+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3224,"log_dups_size":0,"ondisk_log_size":3224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37027840,"num_objects":565,"num_object_clones":0,"num_object_copies":1695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3225,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"51'3199","reported_seq":4251,"reported_epoch":51,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:32.318419+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:32.318419+0000","last_peered":"2024-09-22T03:04:32.318419+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:32.318419+0000","last_undegraded":"2024-09-22T03:04:32.318419+0000","last_fullsized":"2024-09-22T03:04:32.318419+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34734080,"num_objects":530,"num_object_clones":0,"num_object_copies":1590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3200,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":530}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"51'3355","reported_seq":4506,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:33.747424+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:33.747424+0000","last_peered":"2024-09-22T03:04:33.747424+0000","last_clean":"2024-09-22T03:04:33.747424+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:33.747424+0000","last_undegraded":"2024-09-22T03:04:33.747424+0000","last_fullsized":"2024-09-22T03:04:33.747424+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34471936,"num_objects":526,"num_object_clones":0,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3356,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":106,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.204442+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:32.204442+0000","last_peered":"2024-09-22T03:04:32.204442+0000","last_clean":"2024-09-22T03:04:32.204442+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:32.204442+0000","last_undegraded":"2024-09-22T03:04:32.204442+0000","last_fullsized":"2024-09-22T03:04:32.204442+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"51'3340","reported_seq":3472,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:33.693597+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:33.693597+0000","last_peered":"2024-09-22T03:04:33.693597+0000","last_clean":"2024-09-22T03:04:33.693597+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:33.693597+0000","last_undegraded":"2024-09-22T03:04:33.693597+0000","last_fullsized":"2024-09-22T03:04:33.693597+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3340,"log_dups_size":0,"ondisk_log_size":3340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34406400,"num_objects":525,"num_object_clones":0,"num_object_copies":1575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3341,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":135,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.200815+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:32.200815+0000","last_peered":"2024-09-22T03:04:32.200815+0000","last_clean":"2024-09-22T03:04:32.200815+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:32.200815+0000","last_undegraded":"2024-09-22T03:04:32.200815+0000","last_fullsized":"2024-09-22T03:04:32.200815+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"51'3388","reported_seq":4573,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:33.776540+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:33.776540+0000","last_peered":"2024-09-22T03:04:33.776540+0000","last_clean":"2024-09-22T03:04:33.776540+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:33.776540+0000","last_undegraded":"2024-09-22T03:04:33.776540+0000","last_fullsized":"2024-09-22T03:04:33.776540+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3388,"log_dups_size":0,"ondisk_log_size":3388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34668544,"num_objects":529,"num_object_clones":0,"num_object_copies":1587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3389,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":121,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.208762+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:32.208762+0000","last_peered":"2024-09-22T03:04:32.208762+0000","last_clean":"2024-09-22T03:04:32.208762+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:32.208762+0000","last_undegraded":"2024-09-22T03:04:32.208762+0000","last_fullsized":"2024-09-22T03:04:32.208762+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"51'3255","reported_seq":3884,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:34.965604+0000","last_change":"2024-09-22T03:04:25.522373+0000","last_active":"2024-09-22T03:04:34.965604+0000","last_peered":"2024-09-22T03:04:34.965604+0000","last_clean":"2024-09-22T03:02:34.905097+0000","last_became_active":"2024-09-22T03:04:05.614727+0000","last_became_peered":"2024-09-22T03:04:05.614727+0000","last_unstale":"2024-09-22T03:04:34.965604+0000","last_undegraded":"2024-09-22T03:04:34.965604+0000","last_fullsized":"2024-09-22T03:04:05.116295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3255,"log_dups_size":0,"ondisk_log_size":3255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":514,"num_object_clones":0,"num_object_copies":1542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3256,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":116,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.192517+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:32.192517+0000","last_peered":"2024-09-22T03:04:32.192517+0000","last_clean":"2024-09-22T03:04:32.192517+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:32.192517+0000","last_undegraded":"2024-09-22T03:04:32.192517+0000","last_fullsized":"2024-09-22T03:04:32.192517+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":258,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.205266+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:32.205266+0000","last_peered":"2024-09-22T03:04:32.205266+0000","last_clean":"2024-09-22T03:04:32.205266+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:32.205266+0000","last_undegraded":"2024-09-22T03:04:32.205266+0000","last_fullsized":"2024-09-22T03:04:32.205266+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"51'3548","reported_seq":3679,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:34.915118+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:34.915118+0000","last_peered":"2024-09-22T03:04:34.915118+0000","last_clean":"2024-09-22T03:04:34.915118+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:34.915118+0000","last_undegraded":"2024-09-22T03:04:34.915118+0000","last_fullsized":"2024-09-22T03:04:34.915118+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33357824,"num_objects":509,"num_object_clones":0,"num_object_copies":1527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3549,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"51'3363","reported_seq":3469,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:33.790465+0000","last_change":"2024-09-22T03:04:05.634562+0000","last_active":"2024-09-22T03:04:33.790465+0000","last_peered":"2024-09-22T03:04:33.790465+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:33.790465+0000","last_undegraded":"2024-09-22T03:04:05.200849+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3363,"log_dups_size":0,"ondisk_log_size":3363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36700160,"num_objects":560,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3364,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":560}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":89,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.208191+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:32.208191+0000","last_peered":"2024-09-22T03:04:32.208191+0000","last_clean":"2024-09-22T03:04:32.208191+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:32.208191+0000","last_undegraded":"2024-09-22T03:04:32.208191+0000","last_fullsized":"2024-09-22T03:04:32.208191+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"51'3430","reported_seq":3555,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:34.889037+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:34.889037+0000","last_peered":"2024-09-22T03:04:34.889037+0000","last_clean":"2024-09-22T03:04:34.889037+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:34.889037+0000","last_undegraded":"2024-09-22T03:04:34.889037+0000","last_fullsized":"2024-09-22T03:04:34.889037+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35717120,"num_objects":545,"num_object_clones":0,"num_object_copies":1635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3431,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.203331+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:32.203331+0000","last_peered":"2024-09-22T03:04:32.203331+0000","last_clean":"2024-09-22T03:04:32.203331+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:32.203331+0000","last_undegraded":"2024-09-22T03:04:32.203331+0000","last_fullsized":"2024-09-22T03:04:32.203331+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"51'3216","reported_seq":3711,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:32.332356+0000","last_change":"2024-09-22T03:04:32.220527+0000","last_active":"2024-09-22T03:04:32.332356+0000","last_peered":"2024-09-22T03:04:32.332356+0000","last_clean":"2024-09-22T03:02:34.870768+0000","last_became_active":"2024-09-22T03:04:05.621831+0000","last_became_peered":"2024-09-22T03:04:05.621831+0000","last_unstale":"2024-09-22T03:04:32.332356+0000","last_undegraded":"2024-09-22T03:04:32.332356+0000","last_fullsized":"2024-09-22T03:04:05.135956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3216,"log_dups_size":0,"ondisk_log_size":3216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37289984,"num_objects":569,"num_object_clones":0,"num_object_copies":1707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3217,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":569}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.194800+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:32.194800+0000","last_peered":"2024-09-22T03:04:32.194800+0000","last_clean":"2024-09-22T03:04:32.194800+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:32.194800+0000","last_undegraded":"2024-09-22T03:04:32.194800+0000","last_fullsized":"2024-09-22T03:04:32.194800+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"51'3132","reported_seq":3247,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:33.813627+0000","last_change":"2024-09-22T03:04:05.673295+0000","last_active":"2024-09-22T03:04:33.813627+0000","last_peered":"2024-09-22T03:04:33.813627+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:33.813627+0000","last_undegraded":"2024-09-22T03:04:05.671962+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3132,"log_dups_size":0,"ondisk_log_size":3132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3133,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":511}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":98,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.196130+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:32.196130+0000","last_peered":"2024-09-22T03:04:32.196130+0000","last_clean":"2024-09-22T03:04:32.196130+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:32.196130+0000","last_undegraded":"2024-09-22T03:04:32.196130+0000","last_fullsized":"2024-09-22T03:04:32.196130+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":561774616,"num_objects":8573,"num_object_clones":0,"num_object_copies":25719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":52839,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2717,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":115777536,"data_stored":1852244040,"data_compressed":87954453,"data_compressed_allocated":115765248,"data_compressed_original":1852243968,"omap_allocated":0,"internal_metadata":0},"log_size":52826,"ondisk_log_size":52826,"up":48,"acting":41,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70856,"kb_used_data":30072,"kb_used_omap":0,"kb_used_meta":40768,"kb_avail":94300984,"statfs":{"total":96636764160,"available":96564207616,"internally_reserved":0,"allocated":30793728,"data_stored":485980091,"data_compressed":23069256,"data_compressed_allocated":30363648,"data_compressed_original":485818368,"omap_allocated":0,"internal_metadata":41746432},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70108,"kb_used_data":34696,"kb_used_omap":0,"kb_used_meta":35392,"kb_avail":94301732,"statfs":{"total":96636764160,"available":96564973568,"internally_reserved":0,"allocated":35528704,"data_stored":561412547,"data_compressed":26653118,"data_compressed_allocated":35082240,"data_compressed_original":561233920,"omap_allocated":0,"internal_metadata":36241408},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673309,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96580,"kb_used_data":25324,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94275260,"statfs":{"total":96636764160,"available":96537866240,"internally_reserved":0,"allocated":25931776,"data_stored":407989300,"data_compressed":19367926,"data_compressed_allocated":25493504,"data_compressed_original":407814144,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":61788,"kb_used_data":26444,"kb_used_omap":0,"kb_used_meta":35328,"kb_avail":94310052,"statfs":{"total":96636764160,"available":96573493248,"internally_reserved":0,"allocated":27078656,"data_stored":426146267,"data_compressed":20229950,"data_compressed_allocated":26628096,"data_compressed_original":425967616,"omap_allocated":0,"internal_metadata":36175872},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25014272,"data_stored":400162840,"data_compressed":19001871,"data_compressed_allocated":25010176,"data_compressed_original":400162816,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26218496,"data_stored":419430424,"data_compressed":19916799,"data_compressed_allocated":26214400,"data_compressed_original":419430400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34811904,"data_stored":556990464,"data_compressed":26448887,"data_compressed_allocated":34811904,"data_compressed_original":556990464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29732864,"data_stored":475660312,"data_compressed":22586896,"data_compressed_allocated":29728768,"data_compressed_original":475660288,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:37.456 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:04:37.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:37.607+0000 7f7cb224f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:37.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:37.616+0000 7f7cb224f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:38.107 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:38.107 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1921/25731 objects degraded (7.466%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":364,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974243,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+remapped","count":3},{"state_name":"active+recovery_wait","count":1}],"num_pgs":25,"num_pools":3,"num_objects":8577,"data_bytes":562365003,"bytes_used":306515968,"bytes_avail":386240540672,"bytes_total":386547056640,"degraded_objects":1921,"degraded_total":25731,"degraded_ratio":0.074657028487038979,"recovering_objects_per_sec":65,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":405,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":154},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:04:38.126 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:04:38.126 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:38.207 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:38.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:38.257+0000 7f9086618640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:38.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:38.265+0000 7f9086618640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:38.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:38.354+0000 7f04e2317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:38.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:38.361+0000 7f04e2317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:38.543 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:38.543 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:38.561 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-22T03:04:38.355928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539492939,"num_objects":8228,"num_object_clones":0,"num_object_copies":24684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":53327,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2839,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53274,"ondisk_log_size":53274,"up":75,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":296332,"kb_used_data":113540,"kb_used_omap":0,"kb_used_meta":182720,"kb_avail":377191028,"statfs":{"total":386547056640,"available":386243612672,"internally_reserved":0,"allocated":116264960,"data_stored":1832314145,"data_compressed":86983139,"data_compressed_allocated":114491392,"data_compressed_original":1831616512,"omap_allocated":0,"internal_metadata":187105280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-85655552,"num_objects":-1307,"num_object_clones":0,"num_object_copies":-3921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1307,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":527,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001623"},"pg_stats":[{"pgid":"3.f","version":"51'3381","reported_seq":3529,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:37.083321+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:37.083321+0000","last_peered":"2024-09-22T03:04:37.083321+0000","last_clean":"2024-09-22T03:04:37.083321+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:37.083321+0000","last_undegraded":"2024-09-22T03:04:37.083321+0000","last_fullsized":"2024-09-22T03:04:37.083321+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":500,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3382,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"51'3306","reported_seq":3434,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:33.460805+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:33.460805+0000","last_peered":"2024-09-22T03:04:33.460805+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:33.460805+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":519,"num_object_clones":0,"num_object_copies":1557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3307,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":519}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"51'3339","reported_seq":3470,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:33.695222+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:33.695222+0000","last_peered":"2024-09-22T03:04:33.695222+0000","last_clean":"2024-09-22T03:04:33.695222+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:33.695222+0000","last_undegraded":"2024-09-22T03:04:33.695222+0000","last_fullsized":"2024-09-22T03:04:33.695222+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3339,"log_dups_size":0,"ondisk_log_size":3339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751064,"num_objects":516,"num_object_clones":0,"num_object_copies":1548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3340,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"51'3218","reported_seq":3336,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:36.891788+0000","last_change":"2024-09-22T03:04:05.652014+0000","last_active":"2024-09-22T03:04:36.891788+0000","last_peered":"2024-09-22T03:04:36.891788+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:36.891788+0000","last_undegraded":"2024-09-22T03:04:05.225991+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3218,"log_dups_size":0,"ondisk_log_size":3218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"51'3276","reported_seq":3866,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:34.992176+0000","last_change":"2024-09-22T03:04:27.542949+0000","last_active":"2024-09-22T03:04:34.992176+0000","last_peered":"2024-09-22T03:04:34.992176+0000","last_clean":"2024-09-22T03:02:34.806110+0000","last_became_active":"2024-09-22T03:04:05.614390+0000","last_became_peered":"2024-09-22T03:04:05.614390+0000","last_unstale":"2024-09-22T03:04:34.992176+0000","last_undegraded":"2024-09-22T03:04:34.992176+0000","last_fullsized":"2024-09-22T03:04:05.120934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3276,"log_dups_size":0,"ondisk_log_size":3276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34996224,"num_objects":534,"num_object_clones":0,"num_object_copies":1602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":534}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"51'3289","reported_seq":4404,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:36.845717+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:36.845717+0000","last_peered":"2024-09-22T03:04:36.845717+0000","last_clean":"2024-09-22T03:04:36.845717+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:36.845717+0000","last_undegraded":"2024-09-22T03:04:36.845717+0000","last_fullsized":"2024-09-22T03:04:36.845717+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3289,"log_dups_size":0,"ondisk_log_size":3289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":500,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3290,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"51'3274","reported_seq":4326,"reported_epoch":51,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:37.267862+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:37.267862+0000","last_peered":"2024-09-22T03:04:37.267862+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:37.267862+0000","last_undegraded":"2024-09-22T03:04:37.267862+0000","last_fullsized":"2024-09-22T03:04:37.267862+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3274,"log_dups_size":0,"ondisk_log_size":3274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29818880,"num_objects":455,"num_object_clones":0,"num_object_copies":1365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3275,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":455}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"51'3355","reported_seq":4506,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:33.747424+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:33.747424+0000","last_peered":"2024-09-22T03:04:33.747424+0000","last_clean":"2024-09-22T03:04:33.747424+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:33.747424+0000","last_undegraded":"2024-09-22T03:04:33.747424+0000","last_fullsized":"2024-09-22T03:04:33.747424+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34471936,"num_objects":526,"num_object_clones":0,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3356,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":106,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.204442+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:32.204442+0000","last_peered":"2024-09-22T03:04:32.204442+0000","last_clean":"2024-09-22T03:04:32.204442+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:32.204442+0000","last_undegraded":"2024-09-22T03:04:32.204442+0000","last_fullsized":"2024-09-22T03:04:32.204442+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"51'3340","reported_seq":3472,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:33.693597+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:33.693597+0000","last_peered":"2024-09-22T03:04:33.693597+0000","last_clean":"2024-09-22T03:04:33.693597+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:33.693597+0000","last_undegraded":"2024-09-22T03:04:33.693597+0000","last_fullsized":"2024-09-22T03:04:33.693597+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3340,"log_dups_size":0,"ondisk_log_size":3340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34406400,"num_objects":525,"num_object_clones":0,"num_object_copies":1575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3341,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":135,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.200815+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:32.200815+0000","last_peered":"2024-09-22T03:04:32.200815+0000","last_clean":"2024-09-22T03:04:32.200815+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:32.200815+0000","last_undegraded":"2024-09-22T03:04:32.200815+0000","last_fullsized":"2024-09-22T03:04:32.200815+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"51'3388","reported_seq":4573,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:33.776540+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:33.776540+0000","last_peered":"2024-09-22T03:04:33.776540+0000","last_clean":"2024-09-22T03:04:33.776540+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:33.776540+0000","last_undegraded":"2024-09-22T03:04:33.776540+0000","last_fullsized":"2024-09-22T03:04:33.776540+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3388,"log_dups_size":0,"ondisk_log_size":3388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34668544,"num_objects":529,"num_object_clones":0,"num_object_copies":1587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3389,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":121,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.208762+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:32.208762+0000","last_peered":"2024-09-22T03:04:32.208762+0000","last_clean":"2024-09-22T03:04:32.208762+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:32.208762+0000","last_undegraded":"2024-09-22T03:04:32.208762+0000","last_fullsized":"2024-09-22T03:04:32.208762+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"51'3255","reported_seq":3884,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:34.965604+0000","last_change":"2024-09-22T03:04:25.522373+0000","last_active":"2024-09-22T03:04:34.965604+0000","last_peered":"2024-09-22T03:04:34.965604+0000","last_clean":"2024-09-22T03:02:34.905097+0000","last_became_active":"2024-09-22T03:04:05.614727+0000","last_became_peered":"2024-09-22T03:04:05.614727+0000","last_unstale":"2024-09-22T03:04:34.965604+0000","last_undegraded":"2024-09-22T03:04:34.965604+0000","last_fullsized":"2024-09-22T03:04:05.116295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3255,"log_dups_size":0,"ondisk_log_size":3255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":514,"num_object_clones":0,"num_object_copies":1542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3256,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":116,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.192517+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:32.192517+0000","last_peered":"2024-09-22T03:04:32.192517+0000","last_clean":"2024-09-22T03:04:32.192517+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:32.192517+0000","last_undegraded":"2024-09-22T03:04:32.192517+0000","last_fullsized":"2024-09-22T03:04:32.192517+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":258,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.205266+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:32.205266+0000","last_peered":"2024-09-22T03:04:32.205266+0000","last_clean":"2024-09-22T03:04:32.205266+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:32.205266+0000","last_undegraded":"2024-09-22T03:04:32.205266+0000","last_fullsized":"2024-09-22T03:04:32.205266+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"51'3548","reported_seq":3679,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:34.915118+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:34.915118+0000","last_peered":"2024-09-22T03:04:34.915118+0000","last_clean":"2024-09-22T03:04:34.915118+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:34.915118+0000","last_undegraded":"2024-09-22T03:04:34.915118+0000","last_fullsized":"2024-09-22T03:04:34.915118+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33357824,"num_objects":509,"num_object_clones":0,"num_object_copies":1527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3549,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"51'3363","reported_seq":3469,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:33.790465+0000","last_change":"2024-09-22T03:04:05.634562+0000","last_active":"2024-09-22T03:04:33.790465+0000","last_peered":"2024-09-22T03:04:33.790465+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:33.790465+0000","last_undegraded":"2024-09-22T03:04:05.200849+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3363,"log_dups_size":0,"ondisk_log_size":3363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36700160,"num_objects":560,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3364,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":560}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":89,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.208191+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:32.208191+0000","last_peered":"2024-09-22T03:04:32.208191+0000","last_clean":"2024-09-22T03:04:32.208191+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:32.208191+0000","last_undegraded":"2024-09-22T03:04:32.208191+0000","last_fullsized":"2024-09-22T03:04:32.208191+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"51'3430","reported_seq":3555,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:34.889037+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:34.889037+0000","last_peered":"2024-09-22T03:04:34.889037+0000","last_clean":"2024-09-22T03:04:34.889037+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:34.889037+0000","last_undegraded":"2024-09-22T03:04:34.889037+0000","last_fullsized":"2024-09-22T03:04:34.889037+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35717120,"num_objects":545,"num_object_clones":0,"num_object_copies":1635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3431,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.203331+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:32.203331+0000","last_peered":"2024-09-22T03:04:32.203331+0000","last_clean":"2024-09-22T03:04:32.203331+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:32.203331+0000","last_undegraded":"2024-09-22T03:04:32.203331+0000","last_fullsized":"2024-09-22T03:04:32.203331+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"51'3280","reported_seq":4018,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:37.349319+0000","last_change":"2024-09-22T03:04:32.220527+0000","last_active":"2024-09-22T03:04:37.349319+0000","last_peered":"2024-09-22T03:04:37.349319+0000","last_clean":"2024-09-22T03:02:34.870768+0000","last_became_active":"2024-09-22T03:04:05.621831+0000","last_became_peered":"2024-09-22T03:04:05.621831+0000","last_unstale":"2024-09-22T03:04:37.349319+0000","last_undegraded":"2024-09-22T03:04:37.349319+0000","last_fullsized":"2024-09-22T03:04:05.135956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3280,"log_dups_size":0,"ondisk_log_size":3280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33161216,"num_objects":506,"num_object_clones":0,"num_object_copies":1518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3280,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":319,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":506}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.194800+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:32.194800+0000","last_peered":"2024-09-22T03:04:32.194800+0000","last_clean":"2024-09-22T03:04:32.194800+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:32.194800+0000","last_undegraded":"2024-09-22T03:04:32.194800+0000","last_fullsized":"2024-09-22T03:04:32.194800+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"51'3132","reported_seq":3247,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:33.813627+0000","last_change":"2024-09-22T03:04:05.673295+0000","last_active":"2024-09-22T03:04:33.813627+0000","last_peered":"2024-09-22T03:04:33.813627+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:33.813627+0000","last_undegraded":"2024-09-22T03:04:05.671962+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3132,"log_dups_size":0,"ondisk_log_size":3132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3133,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":511}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":98,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.196130+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:32.196130+0000","last_peered":"2024-09-22T03:04:32.196130+0000","last_clean":"2024-09-22T03:04:32.196130+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:32.196130+0000","last_undegraded":"2024-09-22T03:04:32.196130+0000","last_fullsized":"2024-09-22T03:04:32.196130+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":538902552,"num_objects":8224,"num_object_clones":0,"num_object_copies":24672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":53188,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2839,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":112709632,"data_stored":1803157576,"data_compressed":85623566,"data_compressed_allocated":112697344,"data_compressed_original":1803157504,"omap_allocated":0,"internal_metadata":0},"log_size":53174,"ondisk_log_size":53174,"up":48,"acting":41,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70856,"kb_used_data":30072,"kb_used_omap":0,"kb_used_meta":40768,"kb_avail":94300984,"statfs":{"total":96636764160,"available":96564207616,"internally_reserved":0,"allocated":30793728,"data_stored":485980091,"data_compressed":23069256,"data_compressed_allocated":30363648,"data_compressed_original":485818368,"omap_allocated":0,"internal_metadata":41746432},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70108,"kb_used_data":34696,"kb_used_omap":0,"kb_used_meta":35392,"kb_avail":94301732,"statfs":{"total":96636764160,"available":96564973568,"internally_reserved":0,"allocated":35528704,"data_stored":561412547,"data_compressed":26653118,"data_compressed_allocated":35082240,"data_compressed_original":561233920,"omap_allocated":0,"internal_metadata":36241408},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673314,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":93580,"kb_used_data":22328,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94278260,"statfs":{"total":96636764160,"available":96540938240,"internally_reserved":0,"allocated":22863872,"data_stored":358775240,"data_compressed":17030815,"data_compressed_allocated":22417408,"data_compressed_original":358596608,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":61788,"kb_used_data":26444,"kb_used_omap":0,"kb_used_meta":35328,"kb_avail":94310052,"statfs":{"total":96636764160,"available":96573493248,"internally_reserved":0,"allocated":27078656,"data_stored":426146267,"data_compressed":20229950,"data_compressed_allocated":26628096,"data_compressed_original":425967616,"omap_allocated":0,"internal_metadata":36175872},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21946368,"data_stored":351076376,"data_compressed":16670984,"data_compressed_allocated":21942272,"data_compressed_original":351076352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26218496,"data_stored":419430424,"data_compressed":19916799,"data_compressed_allocated":26214400,"data_compressed_original":419430400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34811904,"data_stored":556990464,"data_compressed":26448887,"data_compressed_allocated":34811904,"data_compressed_original":556990464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29732864,"data_stored":475660312,"data_compressed":22586896,"data_compressed_allocated":29728768,"data_compressed_original":475660288,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:40.903 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:41.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:41.033+0000 7f4c4f197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:41.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:41.038+0000 7f4c4f197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:41.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:41.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:41.698+0000 7ff4c80d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:41.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:41.707+0000 7ff4c80d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:42.058 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:42.058 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:42.077 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-22T03:04:40.356339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":491848267,"num_objects":7501,"num_object_clones":0,"num_object_copies":22503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":54054,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3058,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54000,"ondisk_log_size":54000,"up":75,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":293340,"kb_used_data":102228,"kb_used_omap":0,"kb_used_meta":191040,"kb_avail":377194020,"statfs":{"total":386547056640,"available":386246676480,"internally_reserved":0,"allocated":104681472,"data_stored":1646978337,"data_compressed":78182405,"data_compressed_allocated":102907904,"data_compressed_original":1646280704,"omap_allocated":0,"internal_metadata":195624960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-133300224,"num_objects":-2034,"num_object_clones":0,"num_object_copies":-6102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2034,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":746,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.189647"},"pg_stats":[{"pgid":"3.f","version":"51'3381","reported_seq":3529,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:37.083321+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:37.083321+0000","last_peered":"2024-09-22T03:04:37.083321+0000","last_clean":"2024-09-22T03:04:37.083321+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:37.083321+0000","last_undegraded":"2024-09-22T03:04:37.083321+0000","last_fullsized":"2024-09-22T03:04:37.083321+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":500,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3382,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"51'3369","reported_seq":3497,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:38.655522+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:38.655522+0000","last_peered":"2024-09-22T03:04:38.655522+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:38.655522+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3369,"log_dups_size":0,"ondisk_log_size":3369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29884416,"num_objects":456,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3370,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":456}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"51'3408","reported_seq":3539,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:38.663820+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:38.663820+0000","last_peered":"2024-09-22T03:04:38.663820+0000","last_clean":"2024-09-22T03:04:38.663820+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:38.663820+0000","last_undegraded":"2024-09-22T03:04:38.663820+0000","last_fullsized":"2024-09-22T03:04:38.663820+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3408,"log_dups_size":0,"ondisk_log_size":3408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229080,"num_objects":447,"num_object_clones":0,"num_object_copies":1341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3409,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"51'3218","reported_seq":3336,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:36.891788+0000","last_change":"2024-09-22T03:04:05.652014+0000","last_active":"2024-09-22T03:04:36.891788+0000","last_peered":"2024-09-22T03:04:36.891788+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:36.891788+0000","last_undegraded":"2024-09-22T03:04:05.225991+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3218,"log_dups_size":0,"ondisk_log_size":3218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"51'3351","reported_seq":4184,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:40.025490+0000","last_change":"2024-09-22T03:04:27.542949+0000","last_active":"2024-09-22T03:04:40.025490+0000","last_peered":"2024-09-22T03:04:40.025490+0000","last_clean":"2024-09-22T03:02:34.806110+0000","last_became_active":"2024-09-22T03:04:05.614390+0000","last_became_peered":"2024-09-22T03:04:05.614390+0000","last_unstale":"2024-09-22T03:04:40.025490+0000","last_undegraded":"2024-09-22T03:04:40.025490+0000","last_fullsized":"2024-09-22T03:04:05.120934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30015488,"num_objects":458,"num_object_clones":0,"num_object_copies":1374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3352,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"51'3289","reported_seq":4404,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:36.845717+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:36.845717+0000","last_peered":"2024-09-22T03:04:36.845717+0000","last_clean":"2024-09-22T03:04:36.845717+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:36.845717+0000","last_undegraded":"2024-09-22T03:04:36.845717+0000","last_fullsized":"2024-09-22T03:04:36.845717+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3289,"log_dups_size":0,"ondisk_log_size":3289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":500,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3290,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"51'3274","reported_seq":4326,"reported_epoch":51,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:37.267862+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:37.267862+0000","last_peered":"2024-09-22T03:04:37.267862+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:37.267862+0000","last_undegraded":"2024-09-22T03:04:37.267862+0000","last_fullsized":"2024-09-22T03:04:37.267862+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3274,"log_dups_size":0,"ondisk_log_size":3274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29818880,"num_objects":455,"num_object_clones":0,"num_object_copies":1365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3275,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":455}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"51'3422","reported_seq":4573,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:38.630210+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:38.630210+0000","last_peered":"2024-09-22T03:04:38.630210+0000","last_clean":"2024-09-22T03:04:38.630210+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:38.630210+0000","last_undegraded":"2024-09-22T03:04:38.630210+0000","last_fullsized":"2024-09-22T03:04:38.630210+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3422,"log_dups_size":0,"ondisk_log_size":3422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30081024,"num_objects":459,"num_object_clones":0,"num_object_copies":1377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3423,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":106,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.204442+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:32.204442+0000","last_peered":"2024-09-22T03:04:32.204442+0000","last_clean":"2024-09-22T03:04:32.204442+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:32.204442+0000","last_undegraded":"2024-09-22T03:04:32.204442+0000","last_fullsized":"2024-09-22T03:04:32.204442+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"51'3399","reported_seq":3531,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:38.720378+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:38.720378+0000","last_peered":"2024-09-22T03:04:38.720378+0000","last_clean":"2024-09-22T03:04:38.720378+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:38.720378+0000","last_undegraded":"2024-09-22T03:04:38.720378+0000","last_fullsized":"2024-09-22T03:04:38.720378+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3399,"log_dups_size":0,"ondisk_log_size":3399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30539776,"num_objects":466,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3400,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":135,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.200815+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:32.200815+0000","last_peered":"2024-09-22T03:04:32.200815+0000","last_clean":"2024-09-22T03:04:32.200815+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:32.200815+0000","last_undegraded":"2024-09-22T03:04:32.200815+0000","last_fullsized":"2024-09-22T03:04:32.200815+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"51'3449","reported_seq":4634,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:38.223368+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:38.223368+0000","last_peered":"2024-09-22T03:04:38.223368+0000","last_clean":"2024-09-22T03:04:38.223368+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:38.223368+0000","last_undegraded":"2024-09-22T03:04:38.223368+0000","last_fullsized":"2024-09-22T03:04:38.223368+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3449,"log_dups_size":0,"ondisk_log_size":3449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30670848,"num_objects":468,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3450,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":121,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.208762+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:32.208762+0000","last_peered":"2024-09-22T03:04:32.208762+0000","last_clean":"2024-09-22T03:04:32.208762+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:32.208762+0000","last_undegraded":"2024-09-22T03:04:32.208762+0000","last_fullsized":"2024-09-22T03:04:32.208762+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"51'3318","reported_seq":4143,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:39.998431+0000","last_change":"2024-09-22T03:04:25.522373+0000","last_active":"2024-09-22T03:04:39.998431+0000","last_peered":"2024-09-22T03:04:39.998431+0000","last_clean":"2024-09-22T03:02:34.905097+0000","last_became_active":"2024-09-22T03:04:05.614727+0000","last_became_peered":"2024-09-22T03:04:05.614727+0000","last_unstale":"2024-09-22T03:04:39.998431+0000","last_undegraded":"2024-09-22T03:04:39.998431+0000","last_fullsized":"2024-09-22T03:04:05.116295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3318,"log_dups_size":0,"ondisk_log_size":3318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29556736,"num_objects":451,"num_object_clones":0,"num_object_copies":1353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3319,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":353,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":116,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.192517+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:32.192517+0000","last_peered":"2024-09-22T03:04:32.192517+0000","last_clean":"2024-09-22T03:04:32.192517+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:32.192517+0000","last_undegraded":"2024-09-22T03:04:32.192517+0000","last_fullsized":"2024-09-22T03:04:32.192517+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":258,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.205266+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:32.205266+0000","last_peered":"2024-09-22T03:04:32.205266+0000","last_clean":"2024-09-22T03:04:32.205266+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:32.205266+0000","last_undegraded":"2024-09-22T03:04:32.205266+0000","last_fullsized":"2024-09-22T03:04:32.205266+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"51'3614","reported_seq":3745,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:40.015924+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:40.015924+0000","last_peered":"2024-09-22T03:04:40.015924+0000","last_clean":"2024-09-22T03:04:40.015924+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:40.015924+0000","last_undegraded":"2024-09-22T03:04:40.015924+0000","last_fullsized":"2024-09-22T03:04:40.015924+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29032448,"num_objects":443,"num_object_clones":0,"num_object_copies":1329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3615,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"51'3434","reported_seq":3540,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:38.704712+0000","last_change":"2024-09-22T03:04:05.634562+0000","last_active":"2024-09-22T03:04:38.704712+0000","last_peered":"2024-09-22T03:04:38.704712+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:38.704712+0000","last_undegraded":"2024-09-22T03:04:05.200849+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32047104,"num_objects":489,"num_object_clones":0,"num_object_copies":1467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3435,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":489}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":89,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.208191+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:32.208191+0000","last_peered":"2024-09-22T03:04:32.208191+0000","last_clean":"2024-09-22T03:04:32.208191+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:32.208191+0000","last_undegraded":"2024-09-22T03:04:32.208191+0000","last_fullsized":"2024-09-22T03:04:32.208191+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"51'3506","reported_seq":3631,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:39.963460+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:39.963460+0000","last_peered":"2024-09-22T03:04:39.963460+0000","last_clean":"2024-09-22T03:04:39.963460+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:39.963460+0000","last_undegraded":"2024-09-22T03:04:39.963460+0000","last_fullsized":"2024-09-22T03:04:39.963460+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3506,"log_dups_size":0,"ondisk_log_size":3506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30736384,"num_objects":469,"num_object_clones":0,"num_object_copies":1407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.203331+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:32.203331+0000","last_peered":"2024-09-22T03:04:32.203331+0000","last_clean":"2024-09-22T03:04:32.203331+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:32.203331+0000","last_undegraded":"2024-09-22T03:04:32.203331+0000","last_fullsized":"2024-09-22T03:04:32.203331+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"51'3280","reported_seq":4018,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:37.349319+0000","last_change":"2024-09-22T03:04:32.220527+0000","last_active":"2024-09-22T03:04:37.349319+0000","last_peered":"2024-09-22T03:04:37.349319+0000","last_clean":"2024-09-22T03:02:34.870768+0000","last_became_active":"2024-09-22T03:04:05.621831+0000","last_became_peered":"2024-09-22T03:04:05.621831+0000","last_unstale":"2024-09-22T03:04:37.349319+0000","last_undegraded":"2024-09-22T03:04:37.349319+0000","last_fullsized":"2024-09-22T03:04:05.135956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3280,"log_dups_size":0,"ondisk_log_size":3280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33161216,"num_objects":506,"num_object_clones":0,"num_object_copies":1518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3280,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":319,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":506}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.194800+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:32.194800+0000","last_peered":"2024-09-22T03:04:32.194800+0000","last_clean":"2024-09-22T03:04:32.194800+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:32.194800+0000","last_undegraded":"2024-09-22T03:04:32.194800+0000","last_fullsized":"2024-09-22T03:04:32.194800+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"51'3188","reported_seq":3303,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:38.772962+0000","last_change":"2024-09-22T03:04:05.673295+0000","last_active":"2024-09-22T03:04:38.772962+0000","last_peered":"2024-09-22T03:04:38.772962+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:38.772962+0000","last_undegraded":"2024-09-22T03:04:05.671962+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3188,"log_dups_size":0,"ondisk_log_size":3188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29818880,"num_objects":455,"num_object_clones":0,"num_object_copies":1365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3189,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":455}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":98,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.196130+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:32.196130+0000","last_peered":"2024-09-22T03:04:32.196130+0000","last_clean":"2024-09-22T03:04:32.196130+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:32.196130+0000","last_undegraded":"2024-09-22T03:04:32.196130+0000","last_fullsized":"2024-09-22T03:04:32.196130+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":491257880,"num_objects":7497,"num_object_clones":0,"num_object_copies":22491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":53915,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3058,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":101208064,"data_stored":1619132488,"data_compressed":76885072,"data_compressed_allocated":101195776,"data_compressed_original":1619132416,"omap_allocated":0,"internal_metadata":0},"log_size":53900,"ondisk_log_size":53900,"up":48,"acting":41,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71308,"kb_used_data":26360,"kb_used_omap":0,"kb_used_meta":44928,"kb_avail":94300532,"statfs":{"total":96636764160,"available":96563744768,"internally_reserved":0,"allocated":26992640,"data_stored":425162683,"data_compressed":20181320,"data_compressed_allocated":26562560,"data_compressed_original":425000960,"omap_allocated":0,"internal_metadata":46006272},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626467,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69936,"kb_used_data":30368,"kb_used_omap":0,"kb_used_meta":39552,"kb_avail":94301904,"statfs":{"total":96636764160,"available":96565149696,"internally_reserved":0,"allocated":31096832,"data_stored":490502595,"data_compressed":23285935,"data_compressed_allocated":30650368,"data_compressed_original":490323968,"omap_allocated":0,"internal_metadata":40501248},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673314,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":93580,"kb_used_data":22328,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94278260,"statfs":{"total":96636764160,"available":96540938240,"internally_reserved":0,"allocated":22863872,"data_stored":358775240,"data_compressed":17030815,"data_compressed_allocated":22417408,"data_compressed_original":358596608,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":58516,"kb_used_data":23172,"kb_used_omap":0,"kb_used_meta":35328,"kb_avail":94313324,"statfs":{"total":96636764160,"available":96576843776,"internally_reserved":0,"allocated":23728128,"data_stored":372537819,"data_compressed":17684335,"data_compressed_allocated":23277568,"data_compressed_original":372359168,"omap_allocated":0,"internal_metadata":36175872},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21946368,"data_stored":351076376,"data_compressed":16670984,"data_compressed_allocated":21942272,"data_compressed_original":351076352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22884352,"data_stored":366084120,"data_compressed":17383632,"data_compressed_allocated":22880256,"data_compressed_original":366084096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30425088,"data_stored":486801408,"data_compressed":23115936,"data_compressed_allocated":30425088,"data_compressed_original":486801408,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25952256,"data_stored":415170584,"data_compressed":19714520,"data_compressed_allocated":25948160,"data_compressed_original":415170560,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:42.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:04:42.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:42.211+0000 7fdd146c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:42.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:42.219+0000 7fdd146c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:42.707 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:42.707 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1921/22503 objects degraded (8.537%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":368,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974243,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+remapped","count":3},{"state_name":"active+recovery_wait","count":1}],"num_pgs":25,"num_pools":3,"num_objects":7501,"data_bytes":491848267,"bytes_used":300380160,"bytes_avail":386246676480,"bytes_total":386547056640,"degraded_objects":1921,"degraded_total":22503,"degraded_ratio":0.085366395591698882,"recovering_objects_per_sec":73,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":746,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":199},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:04:42.727 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:04:42.727 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:42.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:42.870+0000 7f9f14d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:42.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:42.881+0000 7f9f14d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:43.215 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:43.216 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:43.229 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-22T03:04:42.356701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":491848267,"num_objects":7501,"num_object_clones":0,"num_object_copies":22503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":54054,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3058,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54000,"ondisk_log_size":54000,"up":75,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":293340,"kb_used_data":102228,"kb_used_omap":0,"kb_used_meta":191040,"kb_avail":377194020,"statfs":{"total":386547056640,"available":386246676480,"internally_reserved":0,"allocated":104681472,"data_stored":1646978337,"data_compressed":78182405,"data_compressed_allocated":102907904,"data_compressed_original":1646280704,"omap_allocated":0,"internal_metadata":195624960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-133300224,"num_objects":-2034,"num_object_clones":0,"num_object_copies":-6102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2034,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":746,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.189194"},"pg_stats":[{"pgid":"3.f","version":"51'3381","reported_seq":3529,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:37.083321+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:37.083321+0000","last_peered":"2024-09-22T03:04:37.083321+0000","last_clean":"2024-09-22T03:04:37.083321+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:37.083321+0000","last_undegraded":"2024-09-22T03:04:37.083321+0000","last_fullsized":"2024-09-22T03:04:37.083321+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":500,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3382,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"51'3369","reported_seq":3497,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:38.655522+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:38.655522+0000","last_peered":"2024-09-22T03:04:38.655522+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:38.655522+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3369,"log_dups_size":0,"ondisk_log_size":3369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29884416,"num_objects":456,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3370,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":456}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"51'3408","reported_seq":3539,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:38.663820+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:38.663820+0000","last_peered":"2024-09-22T03:04:38.663820+0000","last_clean":"2024-09-22T03:04:38.663820+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:38.663820+0000","last_undegraded":"2024-09-22T03:04:38.663820+0000","last_fullsized":"2024-09-22T03:04:38.663820+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3408,"log_dups_size":0,"ondisk_log_size":3408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229080,"num_objects":447,"num_object_clones":0,"num_object_copies":1341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3409,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"51'3218","reported_seq":3336,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:36.891788+0000","last_change":"2024-09-22T03:04:05.652014+0000","last_active":"2024-09-22T03:04:36.891788+0000","last_peered":"2024-09-22T03:04:36.891788+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:36.891788+0000","last_undegraded":"2024-09-22T03:04:05.225991+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3218,"log_dups_size":0,"ondisk_log_size":3218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"51'3351","reported_seq":4184,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:40.025490+0000","last_change":"2024-09-22T03:04:27.542949+0000","last_active":"2024-09-22T03:04:40.025490+0000","last_peered":"2024-09-22T03:04:40.025490+0000","last_clean":"2024-09-22T03:02:34.806110+0000","last_became_active":"2024-09-22T03:04:05.614390+0000","last_became_peered":"2024-09-22T03:04:05.614390+0000","last_unstale":"2024-09-22T03:04:40.025490+0000","last_undegraded":"2024-09-22T03:04:40.025490+0000","last_fullsized":"2024-09-22T03:04:05.120934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30015488,"num_objects":458,"num_object_clones":0,"num_object_copies":1374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3352,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"51'3289","reported_seq":4404,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:36.845717+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:36.845717+0000","last_peered":"2024-09-22T03:04:36.845717+0000","last_clean":"2024-09-22T03:04:36.845717+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:36.845717+0000","last_undegraded":"2024-09-22T03:04:36.845717+0000","last_fullsized":"2024-09-22T03:04:36.845717+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3289,"log_dups_size":0,"ondisk_log_size":3289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":500,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3290,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"51'3274","reported_seq":4326,"reported_epoch":51,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:37.267862+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:37.267862+0000","last_peered":"2024-09-22T03:04:37.267862+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:37.267862+0000","last_undegraded":"2024-09-22T03:04:37.267862+0000","last_fullsized":"2024-09-22T03:04:37.267862+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3274,"log_dups_size":0,"ondisk_log_size":3274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29818880,"num_objects":455,"num_object_clones":0,"num_object_copies":1365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3275,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":455}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"51'3422","reported_seq":4573,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:38.630210+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:38.630210+0000","last_peered":"2024-09-22T03:04:38.630210+0000","last_clean":"2024-09-22T03:04:38.630210+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:38.630210+0000","last_undegraded":"2024-09-22T03:04:38.630210+0000","last_fullsized":"2024-09-22T03:04:38.630210+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3422,"log_dups_size":0,"ondisk_log_size":3422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30081024,"num_objects":459,"num_object_clones":0,"num_object_copies":1377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3423,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":106,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.204442+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:32.204442+0000","last_peered":"2024-09-22T03:04:32.204442+0000","last_clean":"2024-09-22T03:04:32.204442+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:32.204442+0000","last_undegraded":"2024-09-22T03:04:32.204442+0000","last_fullsized":"2024-09-22T03:04:32.204442+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"51'3399","reported_seq":3531,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:38.720378+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:38.720378+0000","last_peered":"2024-09-22T03:04:38.720378+0000","last_clean":"2024-09-22T03:04:38.720378+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:38.720378+0000","last_undegraded":"2024-09-22T03:04:38.720378+0000","last_fullsized":"2024-09-22T03:04:38.720378+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3399,"log_dups_size":0,"ondisk_log_size":3399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30539776,"num_objects":466,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3400,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":135,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.200815+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:32.200815+0000","last_peered":"2024-09-22T03:04:32.200815+0000","last_clean":"2024-09-22T03:04:32.200815+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:32.200815+0000","last_undegraded":"2024-09-22T03:04:32.200815+0000","last_fullsized":"2024-09-22T03:04:32.200815+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"51'3449","reported_seq":4634,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:38.223368+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:38.223368+0000","last_peered":"2024-09-22T03:04:38.223368+0000","last_clean":"2024-09-22T03:04:38.223368+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:38.223368+0000","last_undegraded":"2024-09-22T03:04:38.223368+0000","last_fullsized":"2024-09-22T03:04:38.223368+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3449,"log_dups_size":0,"ondisk_log_size":3449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30670848,"num_objects":468,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3450,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":121,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.208762+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:32.208762+0000","last_peered":"2024-09-22T03:04:32.208762+0000","last_clean":"2024-09-22T03:04:32.208762+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:32.208762+0000","last_undegraded":"2024-09-22T03:04:32.208762+0000","last_fullsized":"2024-09-22T03:04:32.208762+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"51'3318","reported_seq":4143,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:39.998431+0000","last_change":"2024-09-22T03:04:25.522373+0000","last_active":"2024-09-22T03:04:39.998431+0000","last_peered":"2024-09-22T03:04:39.998431+0000","last_clean":"2024-09-22T03:02:34.905097+0000","last_became_active":"2024-09-22T03:04:05.614727+0000","last_became_peered":"2024-09-22T03:04:05.614727+0000","last_unstale":"2024-09-22T03:04:39.998431+0000","last_undegraded":"2024-09-22T03:04:39.998431+0000","last_fullsized":"2024-09-22T03:04:05.116295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3318,"log_dups_size":0,"ondisk_log_size":3318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29556736,"num_objects":451,"num_object_clones":0,"num_object_copies":1353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3319,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":353,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":451}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":116,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.192517+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:32.192517+0000","last_peered":"2024-09-22T03:04:32.192517+0000","last_clean":"2024-09-22T03:04:32.192517+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:32.192517+0000","last_undegraded":"2024-09-22T03:04:32.192517+0000","last_fullsized":"2024-09-22T03:04:32.192517+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":258,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.205266+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:32.205266+0000","last_peered":"2024-09-22T03:04:32.205266+0000","last_clean":"2024-09-22T03:04:32.205266+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:32.205266+0000","last_undegraded":"2024-09-22T03:04:32.205266+0000","last_fullsized":"2024-09-22T03:04:32.205266+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"51'3614","reported_seq":3745,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:40.015924+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:40.015924+0000","last_peered":"2024-09-22T03:04:40.015924+0000","last_clean":"2024-09-22T03:04:40.015924+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:40.015924+0000","last_undegraded":"2024-09-22T03:04:40.015924+0000","last_fullsized":"2024-09-22T03:04:40.015924+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29032448,"num_objects":443,"num_object_clones":0,"num_object_copies":1329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3615,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"51'3434","reported_seq":3540,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:38.704712+0000","last_change":"2024-09-22T03:04:05.634562+0000","last_active":"2024-09-22T03:04:38.704712+0000","last_peered":"2024-09-22T03:04:38.704712+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:38.704712+0000","last_undegraded":"2024-09-22T03:04:05.200849+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32047104,"num_objects":489,"num_object_clones":0,"num_object_copies":1467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3435,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":489}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":89,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.208191+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:32.208191+0000","last_peered":"2024-09-22T03:04:32.208191+0000","last_clean":"2024-09-22T03:04:32.208191+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:32.208191+0000","last_undegraded":"2024-09-22T03:04:32.208191+0000","last_fullsized":"2024-09-22T03:04:32.208191+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"51'3506","reported_seq":3631,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:39.963460+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:39.963460+0000","last_peered":"2024-09-22T03:04:39.963460+0000","last_clean":"2024-09-22T03:04:39.963460+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:39.963460+0000","last_undegraded":"2024-09-22T03:04:39.963460+0000","last_fullsized":"2024-09-22T03:04:39.963460+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3506,"log_dups_size":0,"ondisk_log_size":3506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30736384,"num_objects":469,"num_object_clones":0,"num_object_copies":1407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.203331+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:32.203331+0000","last_peered":"2024-09-22T03:04:32.203331+0000","last_clean":"2024-09-22T03:04:32.203331+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:32.203331+0000","last_undegraded":"2024-09-22T03:04:32.203331+0000","last_fullsized":"2024-09-22T03:04:32.203331+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"51'3280","reported_seq":4018,"reported_epoch":51,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:37.349319+0000","last_change":"2024-09-22T03:04:32.220527+0000","last_active":"2024-09-22T03:04:37.349319+0000","last_peered":"2024-09-22T03:04:37.349319+0000","last_clean":"2024-09-22T03:02:34.870768+0000","last_became_active":"2024-09-22T03:04:05.621831+0000","last_became_peered":"2024-09-22T03:04:05.621831+0000","last_unstale":"2024-09-22T03:04:37.349319+0000","last_undegraded":"2024-09-22T03:04:37.349319+0000","last_fullsized":"2024-09-22T03:04:05.135956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3280,"log_dups_size":0,"ondisk_log_size":3280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33161216,"num_objects":506,"num_object_clones":0,"num_object_copies":1518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3280,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":319,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":506}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.194800+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:32.194800+0000","last_peered":"2024-09-22T03:04:32.194800+0000","last_clean":"2024-09-22T03:04:32.194800+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:32.194800+0000","last_undegraded":"2024-09-22T03:04:32.194800+0000","last_fullsized":"2024-09-22T03:04:32.194800+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"51'3188","reported_seq":3303,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:38.772962+0000","last_change":"2024-09-22T03:04:05.673295+0000","last_active":"2024-09-22T03:04:38.772962+0000","last_peered":"2024-09-22T03:04:38.772962+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:38.772962+0000","last_undegraded":"2024-09-22T03:04:05.671962+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3188,"log_dups_size":0,"ondisk_log_size":3188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29818880,"num_objects":455,"num_object_clones":0,"num_object_copies":1365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3189,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":455}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":98,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.196130+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:32.196130+0000","last_peered":"2024-09-22T03:04:32.196130+0000","last_clean":"2024-09-22T03:04:32.196130+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:32.196130+0000","last_undegraded":"2024-09-22T03:04:32.196130+0000","last_fullsized":"2024-09-22T03:04:32.196130+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":491257880,"num_objects":7497,"num_object_clones":0,"num_object_copies":22491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1921,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":53915,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3058,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":101208064,"data_stored":1619132488,"data_compressed":76885072,"data_compressed_allocated":101195776,"data_compressed_original":1619132416,"omap_allocated":0,"internal_metadata":0},"log_size":53900,"ondisk_log_size":53900,"up":48,"acting":41,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71308,"kb_used_data":26360,"kb_used_omap":0,"kb_used_meta":44928,"kb_avail":94300532,"statfs":{"total":96636764160,"available":96563744768,"internally_reserved":0,"allocated":26992640,"data_stored":425162683,"data_compressed":20181320,"data_compressed_allocated":26562560,"data_compressed_original":425000960,"omap_allocated":0,"internal_metadata":46006272},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626467,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69936,"kb_used_data":30368,"kb_used_omap":0,"kb_used_meta":39552,"kb_avail":94301904,"statfs":{"total":96636764160,"available":96565149696,"internally_reserved":0,"allocated":31096832,"data_stored":490502595,"data_compressed":23285935,"data_compressed_allocated":30650368,"data_compressed_original":490323968,"omap_allocated":0,"internal_metadata":40501248},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673314,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":93580,"kb_used_data":22328,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94278260,"statfs":{"total":96636764160,"available":96540938240,"internally_reserved":0,"allocated":22863872,"data_stored":358775240,"data_compressed":17030815,"data_compressed_allocated":22417408,"data_compressed_original":358596608,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":58516,"kb_used_data":23172,"kb_used_omap":0,"kb_used_meta":35328,"kb_avail":94313324,"statfs":{"total":96636764160,"available":96576843776,"internally_reserved":0,"allocated":23728128,"data_stored":372537819,"data_compressed":17684335,"data_compressed_allocated":23277568,"data_compressed_original":372359168,"omap_allocated":0,"internal_metadata":36175872},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21946368,"data_stored":351076376,"data_compressed":16670984,"data_compressed_allocated":21942272,"data_compressed_original":351076352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22884352,"data_stored":366084120,"data_compressed":17383632,"data_compressed_allocated":22880256,"data_compressed_original":366084096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30425088,"data_stored":486801408,"data_compressed":23115936,"data_compressed_allocated":30425088,"data_compressed_original":486801408,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25952256,"data_stored":415170584,"data_compressed":19714520,"data_compressed_allocated":25948160,"data_compressed_original":415170560,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:43.332 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:43.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:43.480+0000 7eff4846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:43.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:43.486+0000 7eff4846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:45.838 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:45.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:45.949+0000 7f1b4d02c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:45.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:45.954+0000 7f1b4d02c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:46.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:46.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:46.347+0000 7f5c4b0d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:46.355+0000 7f5c4b0d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:46.696 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:46.697 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:46.715 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-22T03:04:46.357344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":423494219,"num_objects":6458,"num_object_clones":0,"num_object_copies":19374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":55097,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3427,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55046,"ondisk_log_size":55046,"up":75,"acting":70,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":315524,"kb_used_data":88320,"kb_used_omap":0,"kb_used_meta":227136,"kb_avail":377171836,"statfs":{"total":386547056640,"available":386223960064,"internally_reserved":0,"allocated":90439680,"data_stored":1418726832,"data_compressed":67343309,"data_compressed_allocated":88641536,"data_compressed_original":1418018816,"omap_allocated":0,"internal_metadata":232587264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-68354048,"num_objects":-1043,"num_object_clones":0,"num_object_copies":-3129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1043,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000643"},"pg_stats":[{"pgid":"3.f","version":"51'3445","reported_seq":3593,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:42.070676+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:42.070676+0000","last_peered":"2024-09-22T03:04:42.070676+0000","last_clean":"2024-09-22T03:04:42.070676+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:42.070676+0000","last_undegraded":"2024-09-22T03:04:42.070676+0000","last_fullsized":"2024-09-22T03:04:42.070676+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3445,"log_dups_size":0,"ondisk_log_size":3445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28573696,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3446,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"52'3422","reported_seq":3552,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:43.455098+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:43.455098+0000","last_peered":"2024-09-22T03:04:43.455098+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:43.455098+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3422,"log_dups_size":0,"ondisk_log_size":3422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3423,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":403}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"52'3471","reported_seq":3605,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.737415+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:43.737415+0000","last_peered":"2024-09-22T03:04:43.737415+0000","last_clean":"2024-09-22T03:04:43.737415+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:43.737415+0000","last_undegraded":"2024-09-22T03:04:43.737415+0000","last_fullsized":"2024-09-22T03:04:43.737415+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3471,"log_dups_size":0,"ondisk_log_size":3471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25100312,"num_objects":384,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3472,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"51'3291","reported_seq":3409,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:42.310088+0000","last_change":"2024-09-22T03:04:05.652014+0000","last_active":"2024-09-22T03:04:42.310088+0000","last_peered":"2024-09-22T03:04:42.310088+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:42.310088+0000","last_undegraded":"2024-09-22T03:04:05.225991+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3291,"log_dups_size":0,"ondisk_log_size":3291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26345472,"num_objects":402,"num_object_clones":0,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3292,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":402}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"52'3400","reported_seq":4464,"reported_epoch":53,"state":"peering","last_fresh":"2024-09-22T03:04:44.094998+0000","last_change":"2024-09-22T03:04:44.118100+0000","last_active":"2024-09-22T03:04:44.118100+0000","last_peered":"2024-09-22T03:04:44.094998+0000","last_clean":"2024-09-22T03:02:34.806110+0000","last_became_active":"2024-09-22T03:04:05.614390+0000","last_became_peered":"2024-09-22T03:04:05.614390+0000","last_unstale":"2024-09-22T03:04:44.094998+0000","last_undegraded":"2024-09-22T03:04:44.094998+0000","last_fullsized":"2024-09-22T03:04:05.120934+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3400,"log_dups_size":0,"ondisk_log_size":3400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26869760,"num_objects":410,"num_object_clones":0,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3400,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"51'3360","reported_seq":4475,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:42.325387+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:42.325387+0000","last_peered":"2024-09-22T03:04:42.325387+0000","last_clean":"2024-09-22T03:04:42.325387+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:42.325387+0000","last_undegraded":"2024-09-22T03:04:42.325387+0000","last_fullsized":"2024-09-22T03:04:42.325387+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3360,"log_dups_size":0,"ondisk_log_size":3360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28114944,"num_objects":429,"num_object_clones":0,"num_object_copies":1287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3361,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"51'3341","reported_seq":4393,"reported_epoch":51,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:42.311083+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:42.311083+0000","last_peered":"2024-09-22T03:04:42.311083+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:42.311083+0000","last_undegraded":"2024-09-22T03:04:42.311083+0000","last_fullsized":"2024-09-22T03:04:42.311083+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3341,"log_dups_size":0,"ondisk_log_size":3341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":388,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3342,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":388}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"52'3496","reported_seq":4650,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.543980+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:43.543980+0000","last_peered":"2024-09-22T03:04:43.543980+0000","last_clean":"2024-09-22T03:04:43.543980+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:43.543980+0000","last_undegraded":"2024-09-22T03:04:43.543980+0000","last_fullsized":"2024-09-22T03:04:43.543980+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3496,"log_dups_size":0,"ondisk_log_size":3496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3497,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":112,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-22T03:04:44.119375+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:44.119375+0000","last_peered":"2024-09-22T03:04:44.119375+0000","last_clean":"2024-09-22T03:04:44.119375+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:44.119375+0000","last_undegraded":"2024-09-22T03:04:44.119375+0000","last_fullsized":"2024-09-22T03:04:44.119375+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"52'3469","reported_seq":3604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.707351+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:43.707351+0000","last_peered":"2024-09-22T03:04:43.707351+0000","last_clean":"2024-09-22T03:04:43.707351+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:43.707351+0000","last_undegraded":"2024-09-22T03:04:43.707351+0000","last_fullsized":"2024-09-22T03:04:43.707351+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25952256,"num_objects":396,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3470,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":135,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.200815+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:32.200815+0000","last_peered":"2024-09-22T03:04:32.200815+0000","last_clean":"2024-09-22T03:04:32.200815+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:32.200815+0000","last_undegraded":"2024-09-22T03:04:32.200815+0000","last_fullsized":"2024-09-22T03:04:32.200815+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"52'3512","reported_seq":4700,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.664119+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:43.664119+0000","last_peered":"2024-09-22T03:04:43.664119+0000","last_clean":"2024-09-22T03:04:43.664119+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:43.664119+0000","last_undegraded":"2024-09-22T03:04:43.664119+0000","last_fullsized":"2024-09-22T03:04:43.664119+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3512,"log_dups_size":0,"ondisk_log_size":3512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":405,"num_object_clones":0,"num_object_copies":1215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3513,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.468589+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:43.468589+0000","last_peered":"2024-09-22T03:04:43.468589+0000","last_clean":"2024-09-22T03:04:43.468589+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:43.468589+0000","last_undegraded":"2024-09-22T03:04:43.468589+0000","last_fullsized":"2024-09-22T03:04:43.468589+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"53'3383","reported_seq":4426,"reported_epoch":53,"state":"active+undersized+remapped","last_fresh":"2024-09-22T03:04:44.897146+0000","last_change":"2024-09-22T03:04:44.898283+0000","last_active":"2024-09-22T03:04:44.897146+0000","last_peered":"2024-09-22T03:04:44.897146+0000","last_clean":"2024-09-22T03:02:34.905097+0000","last_became_active":"2024-09-22T03:04:05.614727+0000","last_became_peered":"2024-09-22T03:04:05.614727+0000","last_unstale":"2024-09-22T03:04:44.897146+0000","last_undegraded":"2024-09-22T03:04:44.897146+0000","last_fullsized":"2024-09-22T03:04:05.116295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3383,"log_dups_size":0,"ondisk_log_size":3383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25362432,"num_objects":387,"num_object_clones":0,"num_object_copies":1161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3383,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,3","objects":387}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":122,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-22T03:04:44.115266+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:44.115266+0000","last_peered":"2024-09-22T03:04:44.115266+0000","last_clean":"2024-09-22T03:04:44.115266+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:44.115266+0000","last_undegraded":"2024-09-22T03:04:44.115266+0000","last_fullsized":"2024-09-22T03:04:44.115266+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":258,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.205266+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:32.205266+0000","last_peered":"2024-09-22T03:04:32.205266+0000","last_clean":"2024-09-22T03:04:32.205266+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:32.205266+0000","last_undegraded":"2024-09-22T03:04:32.205266+0000","last_fullsized":"2024-09-22T03:04:32.205266+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"53'3669","reported_seq":3806,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-22T03:04:44.674354+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:44.674354+0000","last_peered":"2024-09-22T03:04:44.674354+0000","last_clean":"2024-09-22T03:04:44.674354+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:44.674354+0000","last_undegraded":"2024-09-22T03:04:44.674354+0000","last_fullsized":"2024-09-22T03:04:44.674354+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":388,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3670,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"52'3503","reported_seq":3633,"reported_epoch":52,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:43.817205+0000","last_change":"2024-09-22T03:04:43.320486+0000","last_active":"2024-09-22T03:04:43.817205+0000","last_peered":"2024-09-22T03:04:43.817205+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:43.817205+0000","last_undegraded":"2024-09-22T03:04:43.817205+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27590656,"num_objects":421,"num_object_clones":0,"num_object_copies":1263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":421}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.446523+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:43.446523+0000","last_peered":"2024-09-22T03:04:43.446523+0000","last_clean":"2024-09-22T03:04:43.446523+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:43.446523+0000","last_undegraded":"2024-09-22T03:04:43.446523+0000","last_fullsized":"2024-09-22T03:04:43.446523+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"53'3569","reported_seq":3700,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-22T03:04:44.665744+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:44.665744+0000","last_peered":"2024-09-22T03:04:44.665744+0000","last_clean":"2024-09-22T03:04:44.665744+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:44.665744+0000","last_undegraded":"2024-09-22T03:04:44.665744+0000","last_fullsized":"2024-09-22T03:04:44.665744+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26607616,"num_objects":406,"num_object_clones":0,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":67,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.453152+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:43.453152+0000","last_peered":"2024-09-22T03:04:43.453152+0000","last_clean":"2024-09-22T03:04:43.453152+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:43.453152+0000","last_undegraded":"2024-09-22T03:04:43.453152+0000","last_fullsized":"2024-09-22T03:04:43.453152+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"51'3364","reported_seq":4381,"reported_epoch":52,"state":"peering","last_fresh":"2024-09-22T03:04:43.401413+0000","last_change":"2024-09-22T03:04:43.577366+0000","last_active":"2024-09-22T03:04:43.577366+0000","last_peered":"2024-09-22T03:04:43.401413+0000","last_clean":"2024-09-22T03:02:34.870768+0000","last_became_active":"2024-09-22T03:04:05.621831+0000","last_became_peered":"2024-09-22T03:04:05.621831+0000","last_unstale":"2024-09-22T03:04:43.401413+0000","last_undegraded":"2024-09-22T03:04:43.401413+0000","last_fullsized":"2024-09-22T03:04:05.135956+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3364,"log_dups_size":0,"ondisk_log_size":3364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27656192,"num_objects":422,"num_object_clones":0,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3364,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.461967+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:43.461967+0000","last_peered":"2024-09-22T03:04:43.461967+0000","last_clean":"2024-09-22T03:04:43.461967+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:43.461967+0000","last_undegraded":"2024-09-22T03:04:43.461967+0000","last_fullsized":"2024-09-22T03:04:43.461967+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"52'3251","reported_seq":3368,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:43.757445+0000","last_change":"2024-09-22T03:04:05.673295+0000","last_active":"2024-09-22T03:04:43.757445+0000","last_peered":"2024-09-22T03:04:43.757445+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:43.757445+0000","last_undegraded":"2024-09-22T03:04:05.671962+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3251,"log_dups_size":0,"ondisk_log_size":3251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":392,"num_object_clones":0,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3252,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":392}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.463147+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:43.463147+0000","last_peered":"2024-09-22T03:04:43.463147+0000","last_clean":"2024-09-22T03:04:43.463147+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:43.463147+0000","last_undegraded":"2024-09-22T03:04:43.463147+0000","last_fullsized":"2024-09-22T03:04:43.463147+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":422903832,"num_objects":6454,"num_object_clones":0,"num_object_copies":19362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":54958,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3427,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87187456,"data_stored":1394802760,"data_compressed":66232696,"data_compressed_allocated":87175168,"data_compressed_original":1394802688,"omap_allocated":0,"internal_metadata":0},"log_size":54946,"ondisk_log_size":54946,"up":48,"acting":43,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73364,"kb_used_data":22852,"kb_used_omap":0,"kb_used_meta":50496,"kb_avail":94298476,"statfs":{"total":96636764160,"available":96561639424,"internally_reserved":0,"allocated":23400448,"data_stored":367432382,"data_compressed":17439648,"data_compressed_allocated":22953984,"data_compressed_original":367263744,"omap_allocated":0,"internal_metadata":51707904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626472,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69876,"kb_used_data":26148,"kb_used_omap":0,"kb_used_meta":43712,"kb_avail":94301964,"statfs":{"total":96636764160,"available":96565211136,"internally_reserved":0,"allocated":26775552,"data_stored":421234511,"data_compressed":19996551,"data_compressed_allocated":26320896,"data_compressed_original":421052416,"omap_allocated":0,"internal_metadata":44761088},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94736,"kb_used_data":19324,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94277104,"statfs":{"total":96636764160,"available":96539754496,"internally_reserved":0,"allocated":19787776,"data_stored":309557704,"data_compressed":14693703,"data_compressed_allocated":19341312,"data_compressed_original":309379072,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77548,"kb_used_data":19996,"kb_used_omap":0,"kb_used_meta":57536,"kb_avail":94294292,"statfs":{"total":96636764160,"available":96557355008,"internally_reserved":0,"allocated":20475904,"data_stored":320502235,"data_compressed":15213407,"data_compressed_allocated":20025344,"data_compressed_original":320323584,"omap_allocated":0,"internal_metadata":58916864},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18952192,"data_stored":303169560,"data_compressed":14396112,"data_compressed_allocated":18948096,"data_compressed_original":303169536,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19689472,"data_stored":314966040,"data_compressed":14956272,"data_compressed_allocated":19685376,"data_compressed_original":314966016,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26103808,"data_stored":417660928,"data_compressed":19832776,"data_compressed_allocated":26103808,"data_compressed_original":417660928,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22441984,"data_stored":359006232,"data_compressed":17047536,"data_compressed_allocated":22437888,"data_compressed_original":359006208,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:46.716 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:04:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:46.843+0000 7fd4f9197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:46.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:46.850+0000 7fd4f9197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:47.240 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:47.240 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1437/19374 objects degraded (7.417%), 3 pgs degraded","count":3},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":373,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974243,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"peering","count":2},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":6458,"data_bytes":423494219,"bytes_used":323096576,"bytes_avail":386223960064,"bytes_total":386547056640,"inactive_pgs_ratio":0.079999998211860657,"degraded_objects":1437,"degraded_total":19374,"degraded_ratio":0.074171570145555893,"recovering_objects_per_sec":92,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":369,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":260},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:04:47.259 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:04:47.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:47.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:47.390+0000 7f38efd57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:47.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:47.398+0000 7f38efd57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:47.719 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:47.719 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:47.738 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-22T03:04:46.357344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":423494219,"num_objects":6458,"num_object_clones":0,"num_object_copies":19374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":55097,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3427,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55046,"ondisk_log_size":55046,"up":75,"acting":70,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":315524,"kb_used_data":88320,"kb_used_omap":0,"kb_used_meta":227136,"kb_avail":377171836,"statfs":{"total":386547056640,"available":386223960064,"internally_reserved":0,"allocated":90439680,"data_stored":1418726832,"data_compressed":67343309,"data_compressed_allocated":88641536,"data_compressed_original":1418018816,"omap_allocated":0,"internal_metadata":232587264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-68354048,"num_objects":-1043,"num_object_clones":0,"num_object_copies":-3129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1043,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000643"},"pg_stats":[{"pgid":"3.f","version":"51'3445","reported_seq":3593,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:42.070676+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:42.070676+0000","last_peered":"2024-09-22T03:04:42.070676+0000","last_clean":"2024-09-22T03:04:42.070676+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:42.070676+0000","last_undegraded":"2024-09-22T03:04:42.070676+0000","last_fullsized":"2024-09-22T03:04:42.070676+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3445,"log_dups_size":0,"ondisk_log_size":3445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28573696,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3446,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"52'3422","reported_seq":3552,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:43.455098+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:43.455098+0000","last_peered":"2024-09-22T03:04:43.455098+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:43.455098+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3422,"log_dups_size":0,"ondisk_log_size":3422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3423,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":403}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"52'3471","reported_seq":3605,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.737415+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:43.737415+0000","last_peered":"2024-09-22T03:04:43.737415+0000","last_clean":"2024-09-22T03:04:43.737415+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:43.737415+0000","last_undegraded":"2024-09-22T03:04:43.737415+0000","last_fullsized":"2024-09-22T03:04:43.737415+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3471,"log_dups_size":0,"ondisk_log_size":3471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25100312,"num_objects":384,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3472,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"51'3291","reported_seq":3409,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:42.310088+0000","last_change":"2024-09-22T03:04:05.652014+0000","last_active":"2024-09-22T03:04:42.310088+0000","last_peered":"2024-09-22T03:04:42.310088+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:42.310088+0000","last_undegraded":"2024-09-22T03:04:05.225991+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3291,"log_dups_size":0,"ondisk_log_size":3291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26345472,"num_objects":402,"num_object_clones":0,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3292,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":402}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"52'3400","reported_seq":4464,"reported_epoch":53,"state":"peering","last_fresh":"2024-09-22T03:04:44.094998+0000","last_change":"2024-09-22T03:04:44.118100+0000","last_active":"2024-09-22T03:04:44.118100+0000","last_peered":"2024-09-22T03:04:44.094998+0000","last_clean":"2024-09-22T03:02:34.806110+0000","last_became_active":"2024-09-22T03:04:05.614390+0000","last_became_peered":"2024-09-22T03:04:05.614390+0000","last_unstale":"2024-09-22T03:04:44.094998+0000","last_undegraded":"2024-09-22T03:04:44.094998+0000","last_fullsized":"2024-09-22T03:04:05.120934+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3400,"log_dups_size":0,"ondisk_log_size":3400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26869760,"num_objects":410,"num_object_clones":0,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3400,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"51'3360","reported_seq":4475,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:42.325387+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:42.325387+0000","last_peered":"2024-09-22T03:04:42.325387+0000","last_clean":"2024-09-22T03:04:42.325387+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:42.325387+0000","last_undegraded":"2024-09-22T03:04:42.325387+0000","last_fullsized":"2024-09-22T03:04:42.325387+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3360,"log_dups_size":0,"ondisk_log_size":3360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28114944,"num_objects":429,"num_object_clones":0,"num_object_copies":1287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3361,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"51'3341","reported_seq":4393,"reported_epoch":51,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:42.311083+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:42.311083+0000","last_peered":"2024-09-22T03:04:42.311083+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:42.311083+0000","last_undegraded":"2024-09-22T03:04:42.311083+0000","last_fullsized":"2024-09-22T03:04:42.311083+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3341,"log_dups_size":0,"ondisk_log_size":3341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":388,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3342,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":388}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"52'3496","reported_seq":4650,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.543980+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:43.543980+0000","last_peered":"2024-09-22T03:04:43.543980+0000","last_clean":"2024-09-22T03:04:43.543980+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:43.543980+0000","last_undegraded":"2024-09-22T03:04:43.543980+0000","last_fullsized":"2024-09-22T03:04:43.543980+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3496,"log_dups_size":0,"ondisk_log_size":3496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3497,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":112,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-22T03:04:44.119375+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:44.119375+0000","last_peered":"2024-09-22T03:04:44.119375+0000","last_clean":"2024-09-22T03:04:44.119375+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:44.119375+0000","last_undegraded":"2024-09-22T03:04:44.119375+0000","last_fullsized":"2024-09-22T03:04:44.119375+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"52'3469","reported_seq":3604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.707351+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:43.707351+0000","last_peered":"2024-09-22T03:04:43.707351+0000","last_clean":"2024-09-22T03:04:43.707351+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:43.707351+0000","last_undegraded":"2024-09-22T03:04:43.707351+0000","last_fullsized":"2024-09-22T03:04:43.707351+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25952256,"num_objects":396,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3470,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":135,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.200815+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:32.200815+0000","last_peered":"2024-09-22T03:04:32.200815+0000","last_clean":"2024-09-22T03:04:32.200815+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:32.200815+0000","last_undegraded":"2024-09-22T03:04:32.200815+0000","last_fullsized":"2024-09-22T03:04:32.200815+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"52'3512","reported_seq":4700,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.664119+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:43.664119+0000","last_peered":"2024-09-22T03:04:43.664119+0000","last_clean":"2024-09-22T03:04:43.664119+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:43.664119+0000","last_undegraded":"2024-09-22T03:04:43.664119+0000","last_fullsized":"2024-09-22T03:04:43.664119+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3512,"log_dups_size":0,"ondisk_log_size":3512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":405,"num_object_clones":0,"num_object_copies":1215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3513,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":124,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.468589+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:43.468589+0000","last_peered":"2024-09-22T03:04:43.468589+0000","last_clean":"2024-09-22T03:04:43.468589+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:43.468589+0000","last_undegraded":"2024-09-22T03:04:43.468589+0000","last_fullsized":"2024-09-22T03:04:43.468589+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"53'3383","reported_seq":4426,"reported_epoch":53,"state":"active+undersized+remapped","last_fresh":"2024-09-22T03:04:44.897146+0000","last_change":"2024-09-22T03:04:44.898283+0000","last_active":"2024-09-22T03:04:44.897146+0000","last_peered":"2024-09-22T03:04:44.897146+0000","last_clean":"2024-09-22T03:02:34.905097+0000","last_became_active":"2024-09-22T03:04:05.614727+0000","last_became_peered":"2024-09-22T03:04:05.614727+0000","last_unstale":"2024-09-22T03:04:44.897146+0000","last_undegraded":"2024-09-22T03:04:44.897146+0000","last_fullsized":"2024-09-22T03:04:05.116295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3383,"log_dups_size":0,"ondisk_log_size":3383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25362432,"num_objects":387,"num_object_clones":0,"num_object_copies":1161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3383,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,3","objects":387}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":122,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-22T03:04:44.115266+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:44.115266+0000","last_peered":"2024-09-22T03:04:44.115266+0000","last_clean":"2024-09-22T03:04:44.115266+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:44.115266+0000","last_undegraded":"2024-09-22T03:04:44.115266+0000","last_fullsized":"2024-09-22T03:04:44.115266+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":258,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:04:32.205266+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:32.205266+0000","last_peered":"2024-09-22T03:04:32.205266+0000","last_clean":"2024-09-22T03:04:32.205266+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:32.205266+0000","last_undegraded":"2024-09-22T03:04:32.205266+0000","last_fullsized":"2024-09-22T03:04:32.205266+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"53'3669","reported_seq":3806,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-22T03:04:44.674354+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:44.674354+0000","last_peered":"2024-09-22T03:04:44.674354+0000","last_clean":"2024-09-22T03:04:44.674354+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:44.674354+0000","last_undegraded":"2024-09-22T03:04:44.674354+0000","last_fullsized":"2024-09-22T03:04:44.674354+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":388,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3670,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"52'3503","reported_seq":3633,"reported_epoch":52,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:43.817205+0000","last_change":"2024-09-22T03:04:43.320486+0000","last_active":"2024-09-22T03:04:43.817205+0000","last_peered":"2024-09-22T03:04:43.817205+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:43.817205+0000","last_undegraded":"2024-09-22T03:04:43.817205+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27590656,"num_objects":421,"num_object_clones":0,"num_object_copies":1263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":421}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.446523+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:43.446523+0000","last_peered":"2024-09-22T03:04:43.446523+0000","last_clean":"2024-09-22T03:04:43.446523+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:43.446523+0000","last_undegraded":"2024-09-22T03:04:43.446523+0000","last_fullsized":"2024-09-22T03:04:43.446523+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"53'3569","reported_seq":3700,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-22T03:04:44.665744+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:44.665744+0000","last_peered":"2024-09-22T03:04:44.665744+0000","last_clean":"2024-09-22T03:04:44.665744+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:44.665744+0000","last_undegraded":"2024-09-22T03:04:44.665744+0000","last_fullsized":"2024-09-22T03:04:44.665744+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26607616,"num_objects":406,"num_object_clones":0,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":67,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.453152+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:43.453152+0000","last_peered":"2024-09-22T03:04:43.453152+0000","last_clean":"2024-09-22T03:04:43.453152+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:43.453152+0000","last_undegraded":"2024-09-22T03:04:43.453152+0000","last_fullsized":"2024-09-22T03:04:43.453152+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"51'3364","reported_seq":4381,"reported_epoch":52,"state":"peering","last_fresh":"2024-09-22T03:04:43.401413+0000","last_change":"2024-09-22T03:04:43.577366+0000","last_active":"2024-09-22T03:04:43.577366+0000","last_peered":"2024-09-22T03:04:43.401413+0000","last_clean":"2024-09-22T03:02:34.870768+0000","last_became_active":"2024-09-22T03:04:05.621831+0000","last_became_peered":"2024-09-22T03:04:05.621831+0000","last_unstale":"2024-09-22T03:04:43.401413+0000","last_undegraded":"2024-09-22T03:04:43.401413+0000","last_fullsized":"2024-09-22T03:04:05.135956+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3364,"log_dups_size":0,"ondisk_log_size":3364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27656192,"num_objects":422,"num_object_clones":0,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3364,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.461967+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:43.461967+0000","last_peered":"2024-09-22T03:04:43.461967+0000","last_clean":"2024-09-22T03:04:43.461967+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:43.461967+0000","last_undegraded":"2024-09-22T03:04:43.461967+0000","last_fullsized":"2024-09-22T03:04:43.461967+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"52'3251","reported_seq":3368,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:43.757445+0000","last_change":"2024-09-22T03:04:05.673295+0000","last_active":"2024-09-22T03:04:43.757445+0000","last_peered":"2024-09-22T03:04:43.757445+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:43.757445+0000","last_undegraded":"2024-09-22T03:04:05.671962+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3251,"log_dups_size":0,"ondisk_log_size":3251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":392,"num_object_clones":0,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3252,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":392}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-22T03:04:43.463147+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:43.463147+0000","last_peered":"2024-09-22T03:04:43.463147+0000","last_clean":"2024-09-22T03:04:43.463147+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:43.463147+0000","last_undegraded":"2024-09-22T03:04:43.463147+0000","last_fullsized":"2024-09-22T03:04:43.463147+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":422903832,"num_objects":6454,"num_object_clones":0,"num_object_copies":19362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":54958,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3427,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87187456,"data_stored":1394802760,"data_compressed":66232696,"data_compressed_allocated":87175168,"data_compressed_original":1394802688,"omap_allocated":0,"internal_metadata":0},"log_size":54946,"ondisk_log_size":54946,"up":48,"acting":43,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73364,"kb_used_data":22852,"kb_used_omap":0,"kb_used_meta":50496,"kb_avail":94298476,"statfs":{"total":96636764160,"available":96561639424,"internally_reserved":0,"allocated":23400448,"data_stored":367432382,"data_compressed":17439648,"data_compressed_allocated":22953984,"data_compressed_original":367263744,"omap_allocated":0,"internal_metadata":51707904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626472,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69876,"kb_used_data":26148,"kb_used_omap":0,"kb_used_meta":43712,"kb_avail":94301964,"statfs":{"total":96636764160,"available":96565211136,"internally_reserved":0,"allocated":26775552,"data_stored":421234511,"data_compressed":19996551,"data_compressed_allocated":26320896,"data_compressed_original":421052416,"omap_allocated":0,"internal_metadata":44761088},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94736,"kb_used_data":19324,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94277104,"statfs":{"total":96636764160,"available":96539754496,"internally_reserved":0,"allocated":19787776,"data_stored":309557704,"data_compressed":14693703,"data_compressed_allocated":19341312,"data_compressed_original":309379072,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77548,"kb_used_data":19996,"kb_used_omap":0,"kb_used_meta":57536,"kb_avail":94294292,"statfs":{"total":96636764160,"available":96557355008,"internally_reserved":0,"allocated":20475904,"data_stored":320502235,"data_compressed":15213407,"data_compressed_allocated":20025344,"data_compressed_original":320323584,"omap_allocated":0,"internal_metadata":58916864},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18952192,"data_stored":303169560,"data_compressed":14396112,"data_compressed_allocated":18948096,"data_compressed_original":303169536,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19689472,"data_stored":314966040,"data_compressed":14956272,"data_compressed_allocated":19685376,"data_compressed_original":314966016,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26103808,"data_stored":417660928,"data_compressed":19832776,"data_compressed_allocated":26103808,"data_compressed_original":417660928,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22441984,"data_stored":359006232,"data_compressed":17047536,"data_compressed_allocated":22437888,"data_compressed_original":359006208,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:48.119 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:48.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:48.256+0000 7fb1ad26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:48.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:48.262+0000 7fb1ad26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:50.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:50.740 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:50.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:50.803+0000 7f8605c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:50.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:50.810+0000 7f8605c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:50.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:50.883+0000 7f6ceff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:50.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:50.895+0000 7f6ceff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:51.248 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:51.248 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:51.265 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-22T03:04:50.358000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":361497163,"num_objects":5512,"num_object_clones":0,"num_object_copies":16536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":56043,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3789,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55988,"ondisk_log_size":55988,"up":75,"acting":71,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":320696,"kb_used_data":75440,"kb_used_omap":0,"kb_used_meta":245184,"kb_avail":377166664,"statfs":{"total":386547056640,"available":386218663936,"internally_reserved":0,"allocated":77250560,"data_stored":1206041449,"data_compressed":57241757,"data_compressed_allocated":75345920,"data_compressed_original":1205288960,"omap_allocated":0,"internal_metadata":251068416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-77070336,"num_objects":-1176,"num_object_clones":0,"num_object_copies":-3528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1176,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.254573"},"pg_stats":[{"pgid":"3.f","version":"55'3504","reported_seq":3664,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:47.268552+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:47.268552+0000","last_peered":"2024-09-22T03:04:47.268552+0000","last_clean":"2024-09-22T03:04:47.268552+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:47.268552+0000","last_undegraded":"2024-09-22T03:04:47.268552+0000","last_fullsized":"2024-09-22T03:04:47.268552+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3504,"log_dups_size":0,"ondisk_log_size":3504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":377,"num_object_clones":0,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3505,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'3474","reported_seq":3610,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:48.681248+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:48.681248+0000","last_peered":"2024-09-22T03:04:48.681248+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:48.681248+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003136,"num_objects":351,"num_object_clones":0,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3475,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":351}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"55'3518","reported_seq":3661,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:48.613939+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:48.613939+0000","last_peered":"2024-09-22T03:04:48.613939+0000","last_clean":"2024-09-22T03:04:48.613939+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:48.613939+0000","last_undegraded":"2024-09-22T03:04:48.613939+0000","last_fullsized":"2024-09-22T03:04:48.613939+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3518,"log_dups_size":0,"ondisk_log_size":3518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020120,"num_objects":337,"num_object_clones":0,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3519,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"55'3342","reported_seq":3627,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:47.348365+0000","last_change":"2024-09-22T03:04:44.918075+0000","last_active":"2024-09-22T03:04:47.348365+0000","last_peered":"2024-09-22T03:04:47.348365+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:47.348365+0000","last_undegraded":"2024-09-22T03:04:47.348365+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3342,"log_dups_size":0,"ondisk_log_size":3342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003136,"num_objects":351,"num_object_clones":0,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":351}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"55'3478","reported_seq":4554,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:49.992587+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:04:49.992587+0000","last_peered":"2024-09-22T03:04:49.992587+0000","last_clean":"2024-09-22T03:04:49.992587+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:04:49.992587+0000","last_undegraded":"2024-09-22T03:04:49.992587+0000","last_fullsized":"2024-09-22T03:04:49.992587+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21692416,"num_objects":331,"num_object_clones":0,"num_object_copies":993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3479,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"55'3420","reported_seq":4547,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:47.114487+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:47.114487+0000","last_peered":"2024-09-22T03:04:47.114487+0000","last_clean":"2024-09-22T03:04:47.114487+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:47.114487+0000","last_undegraded":"2024-09-22T03:04:47.114487+0000","last_fullsized":"2024-09-22T03:04:47.114487+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3420,"log_dups_size":0,"ondisk_log_size":3420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":369,"num_object_clones":0,"num_object_copies":1107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"55'3397","reported_seq":4457,"reported_epoch":55,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:47.124594+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:47.124594+0000","last_peered":"2024-09-22T03:04:47.124594+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:47.124594+0000","last_undegraded":"2024-09-22T03:04:47.124594+0000","last_fullsized":"2024-09-22T03:04:47.124594+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3397,"log_dups_size":0,"ondisk_log_size":3397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":332,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3398,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":332}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"55'3565","reported_seq":4728,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:48.766469+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:48.766469+0000","last_peered":"2024-09-22T03:04:48.766469+0000","last_clean":"2024-09-22T03:04:48.766469+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:48.766469+0000","last_undegraded":"2024-09-22T03:04:48.766469+0000","last_fullsized":"2024-09-22T03:04:48.766469+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3565,"log_dups_size":0,"ondisk_log_size":3565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20709376,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3566,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121580+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:46.121580+0000","last_peered":"2024-09-22T03:04:46.121580+0000","last_clean":"2024-09-22T03:04:46.121580+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:46.121580+0000","last_undegraded":"2024-09-22T03:04:46.121580+0000","last_fullsized":"2024-09-22T03:04:46.121580+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"55'3514","reported_seq":3658,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:48.757507+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:48.757507+0000","last_peered":"2024-09-22T03:04:48.757507+0000","last_clean":"2024-09-22T03:04:48.757507+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:48.757507+0000","last_undegraded":"2024-09-22T03:04:48.757507+0000","last_fullsized":"2024-09-22T03:04:48.757507+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003136,"num_objects":351,"num_object_clones":0,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":147,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121377+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:46.121377+0000","last_peered":"2024-09-22T03:04:46.121377+0000","last_clean":"2024-09-22T03:04:46.121377+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:46.121377+0000","last_undegraded":"2024-09-22T03:04:46.121377+0000","last_fullsized":"2024-09-22T03:04:46.121377+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'3569","reported_seq":4766,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:48.730831+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:48.730831+0000","last_peered":"2024-09-22T03:04:48.730831+0000","last_clean":"2024-09-22T03:04:48.730831+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:48.730831+0000","last_undegraded":"2024-09-22T03:04:48.730831+0000","last_fullsized":"2024-09-22T03:04:48.730831+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22806528,"num_objects":348,"num_object_clones":0,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":133,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.127168+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:46.127168+0000","last_peered":"2024-09-22T03:04:46.127168+0000","last_clean":"2024-09-22T03:04:46.127168+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:46.127168+0000","last_undegraded":"2024-09-22T03:04:46.127168+0000","last_fullsized":"2024-09-22T03:04:46.127168+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"55'3441","reported_seq":4494,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:49.742830+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:04:49.742830+0000","last_peered":"2024-09-22T03:04:49.742830+0000","last_clean":"2024-09-22T03:04:49.742830+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:04:49.742830+0000","last_undegraded":"2024-09-22T03:04:49.742830+0000","last_fullsized":"2024-09-22T03:04:49.742830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3441,"log_dups_size":0,"ondisk_log_size":3441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":328,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":128,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.120371+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:46.120371+0000","last_peered":"2024-09-22T03:04:46.120371+0000","last_clean":"2024-09-22T03:04:46.120371+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:46.120371+0000","last_undegraded":"2024-09-22T03:04:46.120371+0000","last_fullsized":"2024-09-22T03:04:46.120371+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":270,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.126224+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:46.126224+0000","last_peered":"2024-09-22T03:04:46.126224+0000","last_clean":"2024-09-22T03:04:46.126224+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:46.126224+0000","last_undegraded":"2024-09-22T03:04:46.126224+0000","last_fullsized":"2024-09-22T03:04:46.126224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"55'3736","reported_seq":3879,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:49.838845+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:49.838845+0000","last_peered":"2024-09-22T03:04:49.838845+0000","last_clean":"2024-09-22T03:04:49.838845+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:49.838845+0000","last_undegraded":"2024-09-22T03:04:49.838845+0000","last_fullsized":"2024-09-22T03:04:49.838845+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21037056,"num_objects":321,"num_object_clones":0,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"55'3565","reported_seq":3942,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:48.797710+0000","last_change":"2024-09-22T03:04:43.320486+0000","last_active":"2024-09-22T03:04:48.797710+0000","last_peered":"2024-09-22T03:04:48.797710+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:48.797710+0000","last_undegraded":"2024-09-22T03:04:48.797710+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3565,"log_dups_size":0,"ondisk_log_size":3565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23461888,"num_objects":358,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3566,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":358}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":101,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.139422+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:46.139422+0000","last_peered":"2024-09-22T03:04:46.139422+0000","last_clean":"2024-09-22T03:04:46.139422+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:46.139422+0000","last_undegraded":"2024-09-22T03:04:46.139422+0000","last_fullsized":"2024-09-22T03:04:46.139422+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"55'3623","reported_seq":3760,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:50.028562+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:50.028562+0000","last_peered":"2024-09-22T03:04:50.028562+0000","last_clean":"2024-09-22T03:04:50.028562+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:50.028562+0000","last_undegraded":"2024-09-22T03:04:50.028562+0000","last_fullsized":"2024-09-22T03:04:50.028562+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":352,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3624,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121224+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:46.121224+0000","last_peered":"2024-09-22T03:04:46.121224+0000","last_clean":"2024-09-22T03:04:46.121224+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:46.121224+0000","last_undegraded":"2024-09-22T03:04:46.121224+0000","last_fullsized":"2024-09-22T03:04:46.121224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"55'3427","reported_seq":4464,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:48.760673+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:04:48.760673+0000","last_peered":"2024-09-22T03:04:48.760673+0000","last_clean":"2024-09-22T03:04:48.760673+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:04:48.760673+0000","last_undegraded":"2024-09-22T03:04:48.760673+0000","last_fullsized":"2024-09-22T03:04:48.760673+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3427,"log_dups_size":0,"ondisk_log_size":3427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23461888,"num_objects":358,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3428,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":108,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.138950+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:46.138950+0000","last_peered":"2024-09-22T03:04:46.138950+0000","last_clean":"2024-09-22T03:04:46.138950+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:46.138950+0000","last_undegraded":"2024-09-22T03:04:46.138950+0000","last_fullsized":"2024-09-22T03:04:46.138950+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"55'3315","reported_seq":3765,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:48.785402+0000","last_change":"2024-09-22T03:04:44.549017+0000","last_active":"2024-09-22T03:04:48.785402+0000","last_peered":"2024-09-22T03:04:48.785402+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:48.785402+0000","last_undegraded":"2024-09-22T03:04:48.785402+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3315,"log_dups_size":0,"ondisk_log_size":3315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":328,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3316,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":328}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":110,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.140783+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:46.140783+0000","last_peered":"2024-09-22T03:04:46.140783+0000","last_clean":"2024-09-22T03:04:46.140783+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:46.140783+0000","last_undegraded":"2024-09-22T03:04:46.140783+0000","last_fullsized":"2024-09-22T03:04:46.140783+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":360906776,"num_objects":5508,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":55904,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3789,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73666560,"data_stored":1178468424,"data_compressed":55959984,"data_compressed_allocated":73654272,"data_compressed_original":1178468352,"omap_allocated":0,"internal_metadata":0},"log_size":55888,"ondisk_log_size":55888,"up":48,"acting":44,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74088,"kb_used_data":19416,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94297752,"statfs":{"total":96636764160,"available":96560898048,"internally_reserved":0,"allocated":19881984,"data_stored":310881613,"data_compressed":14753992,"data_compressed_allocated":19419136,"data_compressed_original":310706176,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626477,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75204,"kb_used_data":21748,"kb_used_omap":0,"kb_used_meta":53440,"kb_avail":94296636,"statfs":{"total":96636764160,"available":96559755264,"internally_reserved":0,"allocated":22269952,"data_stored":348761941,"data_compressed":16554679,"data_compressed_allocated":21790720,"data_compressed_original":348569600,"omap_allocated":0,"internal_metadata":54722560},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673324,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":92408,"kb_used_data":17000,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94279432,"statfs":{"total":96636764160,"available":96542138368,"internally_reserved":0,"allocated":17408000,"data_stored":270970714,"data_compressed":12860735,"data_compressed_allocated":16928768,"data_compressed_original":270778368,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888218,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78996,"kb_used_data":17276,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94292844,"statfs":{"total":96636764160,"available":96555872256,"internally_reserved":0,"allocated":17690624,"data_stored":275427181,"data_compressed":13072351,"data_compressed_allocated":17207296,"data_compressed_original":275234816,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16470016,"data_stored":263454744,"data_compressed":12510240,"data_compressed_allocated":16465920,"data_compressed_original":263454720,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16826368,"data_stored":269156376,"data_compressed":12780984,"data_compressed_allocated":16822272,"data_compressed_original":269156352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21581824,"data_stored":345309184,"data_compressed":16397128,"data_compressed_allocated":21581824,"data_compressed_original":345309184,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18788352,"data_stored":300548120,"data_compressed":14271632,"data_compressed_allocated":18784256,"data_compressed_original":300548096,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:51.266 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:04:51.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:51.410+0000 7f8217117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:51.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:51.418+0000 7f8217117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:51.810 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:51.810 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 488/16536 objects degraded (2.951%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":377,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974243,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovering+undersized+remapped","count":3},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":5512,"data_bytes":361497163,"bytes_used":328392704,"bytes_avail":386218663936,"bytes_total":386547056640,"degraded_objects":488,"degraded_total":16536,"degraded_ratio":0.029511369134010642,"recovering_objects_per_sec":93,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":583,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":188},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:04:51.826 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:04:51.826 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:51.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:51.965+0000 7f03756d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:51.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:51.977+0000 7f03756d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:52.274 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:52.274 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:52.290 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-22T03:04:50.358000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":361497163,"num_objects":5512,"num_object_clones":0,"num_object_copies":16536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":56043,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3789,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55988,"ondisk_log_size":55988,"up":75,"acting":71,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":320696,"kb_used_data":75440,"kb_used_omap":0,"kb_used_meta":245184,"kb_avail":377166664,"statfs":{"total":386547056640,"available":386218663936,"internally_reserved":0,"allocated":77250560,"data_stored":1206041449,"data_compressed":57241757,"data_compressed_allocated":75345920,"data_compressed_original":1205288960,"omap_allocated":0,"internal_metadata":251068416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-77070336,"num_objects":-1176,"num_object_clones":0,"num_object_copies":-3528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1176,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.254573"},"pg_stats":[{"pgid":"3.f","version":"55'3504","reported_seq":3664,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:47.268552+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:47.268552+0000","last_peered":"2024-09-22T03:04:47.268552+0000","last_clean":"2024-09-22T03:04:47.268552+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:47.268552+0000","last_undegraded":"2024-09-22T03:04:47.268552+0000","last_fullsized":"2024-09-22T03:04:47.268552+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3504,"log_dups_size":0,"ondisk_log_size":3504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":377,"num_object_clones":0,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3505,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'3474","reported_seq":3610,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:48.681248+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:48.681248+0000","last_peered":"2024-09-22T03:04:48.681248+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:48.681248+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003136,"num_objects":351,"num_object_clones":0,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3475,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":351}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"55'3518","reported_seq":3661,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:48.613939+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:48.613939+0000","last_peered":"2024-09-22T03:04:48.613939+0000","last_clean":"2024-09-22T03:04:48.613939+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:48.613939+0000","last_undegraded":"2024-09-22T03:04:48.613939+0000","last_fullsized":"2024-09-22T03:04:48.613939+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3518,"log_dups_size":0,"ondisk_log_size":3518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020120,"num_objects":337,"num_object_clones":0,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3519,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"55'3342","reported_seq":3627,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:47.348365+0000","last_change":"2024-09-22T03:04:44.918075+0000","last_active":"2024-09-22T03:04:47.348365+0000","last_peered":"2024-09-22T03:04:47.348365+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:47.348365+0000","last_undegraded":"2024-09-22T03:04:47.348365+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3342,"log_dups_size":0,"ondisk_log_size":3342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003136,"num_objects":351,"num_object_clones":0,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":351}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"55'3478","reported_seq":4554,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:49.992587+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:04:49.992587+0000","last_peered":"2024-09-22T03:04:49.992587+0000","last_clean":"2024-09-22T03:04:49.992587+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:04:49.992587+0000","last_undegraded":"2024-09-22T03:04:49.992587+0000","last_fullsized":"2024-09-22T03:04:49.992587+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21692416,"num_objects":331,"num_object_clones":0,"num_object_copies":993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3479,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"55'3420","reported_seq":4547,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:47.114487+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:47.114487+0000","last_peered":"2024-09-22T03:04:47.114487+0000","last_clean":"2024-09-22T03:04:47.114487+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:47.114487+0000","last_undegraded":"2024-09-22T03:04:47.114487+0000","last_fullsized":"2024-09-22T03:04:47.114487+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3420,"log_dups_size":0,"ondisk_log_size":3420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":369,"num_object_clones":0,"num_object_copies":1107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"55'3397","reported_seq":4457,"reported_epoch":55,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:47.124594+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:47.124594+0000","last_peered":"2024-09-22T03:04:47.124594+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:47.124594+0000","last_undegraded":"2024-09-22T03:04:47.124594+0000","last_fullsized":"2024-09-22T03:04:47.124594+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3397,"log_dups_size":0,"ondisk_log_size":3397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":332,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3398,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":332}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"55'3565","reported_seq":4728,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:48.766469+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:48.766469+0000","last_peered":"2024-09-22T03:04:48.766469+0000","last_clean":"2024-09-22T03:04:48.766469+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:48.766469+0000","last_undegraded":"2024-09-22T03:04:48.766469+0000","last_fullsized":"2024-09-22T03:04:48.766469+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3565,"log_dups_size":0,"ondisk_log_size":3565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20709376,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3566,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121580+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:46.121580+0000","last_peered":"2024-09-22T03:04:46.121580+0000","last_clean":"2024-09-22T03:04:46.121580+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:46.121580+0000","last_undegraded":"2024-09-22T03:04:46.121580+0000","last_fullsized":"2024-09-22T03:04:46.121580+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"55'3514","reported_seq":3658,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:48.757507+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:48.757507+0000","last_peered":"2024-09-22T03:04:48.757507+0000","last_clean":"2024-09-22T03:04:48.757507+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:48.757507+0000","last_undegraded":"2024-09-22T03:04:48.757507+0000","last_fullsized":"2024-09-22T03:04:48.757507+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003136,"num_objects":351,"num_object_clones":0,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":147,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121377+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:46.121377+0000","last_peered":"2024-09-22T03:04:46.121377+0000","last_clean":"2024-09-22T03:04:46.121377+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:46.121377+0000","last_undegraded":"2024-09-22T03:04:46.121377+0000","last_fullsized":"2024-09-22T03:04:46.121377+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'3569","reported_seq":4766,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:48.730831+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:48.730831+0000","last_peered":"2024-09-22T03:04:48.730831+0000","last_clean":"2024-09-22T03:04:48.730831+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:48.730831+0000","last_undegraded":"2024-09-22T03:04:48.730831+0000","last_fullsized":"2024-09-22T03:04:48.730831+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22806528,"num_objects":348,"num_object_clones":0,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":133,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.127168+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:46.127168+0000","last_peered":"2024-09-22T03:04:46.127168+0000","last_clean":"2024-09-22T03:04:46.127168+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:46.127168+0000","last_undegraded":"2024-09-22T03:04:46.127168+0000","last_fullsized":"2024-09-22T03:04:46.127168+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"55'3441","reported_seq":4494,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:49.742830+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:04:49.742830+0000","last_peered":"2024-09-22T03:04:49.742830+0000","last_clean":"2024-09-22T03:04:49.742830+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:04:49.742830+0000","last_undegraded":"2024-09-22T03:04:49.742830+0000","last_fullsized":"2024-09-22T03:04:49.742830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3441,"log_dups_size":0,"ondisk_log_size":3441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":328,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":128,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.120371+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:46.120371+0000","last_peered":"2024-09-22T03:04:46.120371+0000","last_clean":"2024-09-22T03:04:46.120371+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:46.120371+0000","last_undegraded":"2024-09-22T03:04:46.120371+0000","last_fullsized":"2024-09-22T03:04:46.120371+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":270,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.126224+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:46.126224+0000","last_peered":"2024-09-22T03:04:46.126224+0000","last_clean":"2024-09-22T03:04:46.126224+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:46.126224+0000","last_undegraded":"2024-09-22T03:04:46.126224+0000","last_fullsized":"2024-09-22T03:04:46.126224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"55'3736","reported_seq":3879,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:49.838845+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:49.838845+0000","last_peered":"2024-09-22T03:04:49.838845+0000","last_clean":"2024-09-22T03:04:49.838845+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:49.838845+0000","last_undegraded":"2024-09-22T03:04:49.838845+0000","last_fullsized":"2024-09-22T03:04:49.838845+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21037056,"num_objects":321,"num_object_clones":0,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"55'3565","reported_seq":3942,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:48.797710+0000","last_change":"2024-09-22T03:04:43.320486+0000","last_active":"2024-09-22T03:04:48.797710+0000","last_peered":"2024-09-22T03:04:48.797710+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:48.797710+0000","last_undegraded":"2024-09-22T03:04:48.797710+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3565,"log_dups_size":0,"ondisk_log_size":3565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23461888,"num_objects":358,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3566,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":358}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":101,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.139422+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:46.139422+0000","last_peered":"2024-09-22T03:04:46.139422+0000","last_clean":"2024-09-22T03:04:46.139422+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:46.139422+0000","last_undegraded":"2024-09-22T03:04:46.139422+0000","last_fullsized":"2024-09-22T03:04:46.139422+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"55'3623","reported_seq":3760,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:50.028562+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:50.028562+0000","last_peered":"2024-09-22T03:04:50.028562+0000","last_clean":"2024-09-22T03:04:50.028562+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:50.028562+0000","last_undegraded":"2024-09-22T03:04:50.028562+0000","last_fullsized":"2024-09-22T03:04:50.028562+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":352,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3624,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121224+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:46.121224+0000","last_peered":"2024-09-22T03:04:46.121224+0000","last_clean":"2024-09-22T03:04:46.121224+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:46.121224+0000","last_undegraded":"2024-09-22T03:04:46.121224+0000","last_fullsized":"2024-09-22T03:04:46.121224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"55'3427","reported_seq":4464,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:48.760673+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:04:48.760673+0000","last_peered":"2024-09-22T03:04:48.760673+0000","last_clean":"2024-09-22T03:04:48.760673+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:04:48.760673+0000","last_undegraded":"2024-09-22T03:04:48.760673+0000","last_fullsized":"2024-09-22T03:04:48.760673+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3427,"log_dups_size":0,"ondisk_log_size":3427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23461888,"num_objects":358,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3428,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":108,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.138950+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:46.138950+0000","last_peered":"2024-09-22T03:04:46.138950+0000","last_clean":"2024-09-22T03:04:46.138950+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:46.138950+0000","last_undegraded":"2024-09-22T03:04:46.138950+0000","last_fullsized":"2024-09-22T03:04:46.138950+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"55'3315","reported_seq":3765,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:48.785402+0000","last_change":"2024-09-22T03:04:44.549017+0000","last_active":"2024-09-22T03:04:48.785402+0000","last_peered":"2024-09-22T03:04:48.785402+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:48.785402+0000","last_undegraded":"2024-09-22T03:04:48.785402+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3315,"log_dups_size":0,"ondisk_log_size":3315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":328,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3316,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":328}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":110,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.140783+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:46.140783+0000","last_peered":"2024-09-22T03:04:46.140783+0000","last_clean":"2024-09-22T03:04:46.140783+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:46.140783+0000","last_undegraded":"2024-09-22T03:04:46.140783+0000","last_fullsized":"2024-09-22T03:04:46.140783+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":360906776,"num_objects":5508,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":55904,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3789,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73666560,"data_stored":1178468424,"data_compressed":55959984,"data_compressed_allocated":73654272,"data_compressed_original":1178468352,"omap_allocated":0,"internal_metadata":0},"log_size":55888,"ondisk_log_size":55888,"up":48,"acting":44,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74088,"kb_used_data":19416,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94297752,"statfs":{"total":96636764160,"available":96560898048,"internally_reserved":0,"allocated":19881984,"data_stored":310881613,"data_compressed":14753992,"data_compressed_allocated":19419136,"data_compressed_original":310706176,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626477,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75204,"kb_used_data":21748,"kb_used_omap":0,"kb_used_meta":53440,"kb_avail":94296636,"statfs":{"total":96636764160,"available":96559755264,"internally_reserved":0,"allocated":22269952,"data_stored":348761941,"data_compressed":16554679,"data_compressed_allocated":21790720,"data_compressed_original":348569600,"omap_allocated":0,"internal_metadata":54722560},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673324,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":92408,"kb_used_data":17000,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94279432,"statfs":{"total":96636764160,"available":96542138368,"internally_reserved":0,"allocated":17408000,"data_stored":270970714,"data_compressed":12860735,"data_compressed_allocated":16928768,"data_compressed_original":270778368,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888218,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78996,"kb_used_data":17276,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94292844,"statfs":{"total":96636764160,"available":96555872256,"internally_reserved":0,"allocated":17690624,"data_stored":275427181,"data_compressed":13072351,"data_compressed_allocated":17207296,"data_compressed_original":275234816,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16470016,"data_stored":263454744,"data_compressed":12510240,"data_compressed_allocated":16465920,"data_compressed_original":263454720,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16826368,"data_stored":269156376,"data_compressed":12780984,"data_compressed_allocated":16822272,"data_compressed_original":269156352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21581824,"data_stored":345309184,"data_compressed":16397128,"data_compressed_allocated":21581824,"data_compressed_original":345309184,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18788352,"data_stored":300548120,"data_compressed":14271632,"data_compressed_allocated":18784256,"data_compressed_original":300548096,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:53.109 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:53.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:53.261+0000 7fdcb9f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:53.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:53.268+0000 7fdcb9f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:55.292 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:55.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:55.425+0000 7fa76d6b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:55.432+0000 7fa76d6b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:55.485 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:04:55.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:55.624+0000 7fb7ba957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:55.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:55.630+0000 7fb7ba957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:55.703 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:55.703 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:55.716 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-22T03:04:54.358698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":307429963,"num_objects":4687,"num_object_clones":0,"num_object_copies":14061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":56868,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4163,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56815,"ondisk_log_size":56815,"up":75,"acting":71,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":301588,"kb_used_data":64652,"kb_used_omap":0,"kb_used_meta":236864,"kb_avail":377185772,"statfs":{"total":386547056640,"available":386238230528,"internally_reserved":0,"allocated":66203648,"data_stored":1029290857,"data_compressed":48848693,"data_compressed_allocated":64299008,"data_compressed_original":1028538368,"omap_allocated":0,"internal_metadata":242548736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-116064256,"num_objects":-1771,"num_object_clones":0,"num_object_copies":-5313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1771,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.248621"},"pg_stats":[{"pgid":"3.f","version":"55'3582","reported_seq":3742,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:52.252914+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:52.252914+0000","last_peered":"2024-09-22T03:04:52.252914+0000","last_clean":"2024-09-22T03:04:52.252914+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:52.252914+0000","last_undegraded":"2024-09-22T03:04:52.252914+0000","last_fullsized":"2024-09-22T03:04:52.252914+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19595264,"num_objects":299,"num_object_clones":0,"num_object_copies":897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3583,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'3543","reported_seq":3679,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:53.746204+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:53.746204+0000","last_peered":"2024-09-22T03:04:53.746204+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:53.746204+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3543,"log_dups_size":0,"ondisk_log_size":3543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18481152,"num_objects":282,"num_object_clones":0,"num_object_copies":846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3544,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":282}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"55'3585","reported_seq":3728,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:53.734194+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:53.734194+0000","last_peered":"2024-09-22T03:04:53.734194+0000","last_clean":"2024-09-22T03:04:53.734194+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:53.734194+0000","last_undegraded":"2024-09-22T03:04:53.734194+0000","last_fullsized":"2024-09-22T03:04:53.734194+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17629208,"num_objects":270,"num_object_clones":0,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3586,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"55'3411","reported_seq":3951,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:52.334618+0000","last_change":"2024-09-22T03:04:44.918075+0000","last_active":"2024-09-22T03:04:52.334618+0000","last_peered":"2024-09-22T03:04:52.334618+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:52.334618+0000","last_undegraded":"2024-09-22T03:04:52.334618+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18546688,"num_objects":283,"num_object_clones":0,"num_object_copies":849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":283}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"55'3478","reported_seq":4554,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:49.992587+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:04:49.992587+0000","last_peered":"2024-09-22T03:04:49.992587+0000","last_clean":"2024-09-22T03:04:49.992587+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:04:49.992587+0000","last_undegraded":"2024-09-22T03:04:49.992587+0000","last_fullsized":"2024-09-22T03:04:49.992587+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21692416,"num_objects":331,"num_object_clones":0,"num_object_copies":993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3479,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"55'3486","reported_seq":4613,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:52.263897+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:52.263897+0000","last_peered":"2024-09-22T03:04:52.263897+0000","last_clean":"2024-09-22T03:04:52.263897+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:52.263897+0000","last_undegraded":"2024-09-22T03:04:52.263897+0000","last_fullsized":"2024-09-22T03:04:52.263897+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3486,"log_dups_size":0,"ondisk_log_size":3486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3487,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"55'3458","reported_seq":4518,"reported_epoch":55,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:52.309987+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:52.309987+0000","last_peered":"2024-09-22T03:04:52.309987+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:52.309987+0000","last_undegraded":"2024-09-22T03:04:52.309987+0000","last_fullsized":"2024-09-22T03:04:52.309987+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17760256,"num_objects":271,"num_object_clones":0,"num_object_copies":813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3459,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":271}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"55'3638","reported_seq":4801,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:53.732957+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:53.732957+0000","last_peered":"2024-09-22T03:04:53.732957+0000","last_clean":"2024-09-22T03:04:53.732957+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:53.732957+0000","last_undegraded":"2024-09-22T03:04:53.732957+0000","last_fullsized":"2024-09-22T03:04:53.732957+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3638,"log_dups_size":0,"ondisk_log_size":3638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15925248,"num_objects":243,"num_object_clones":0,"num_object_copies":729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3639,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121580+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:46.121580+0000","last_peered":"2024-09-22T03:04:46.121580+0000","last_clean":"2024-09-22T03:04:46.121580+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:46.121580+0000","last_undegraded":"2024-09-22T03:04:46.121580+0000","last_fullsized":"2024-09-22T03:04:46.121580+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"55'3596","reported_seq":3740,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:53.772740+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:53.772740+0000","last_peered":"2024-09-22T03:04:53.772740+0000","last_clean":"2024-09-22T03:04:53.772740+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:53.772740+0000","last_undegraded":"2024-09-22T03:04:53.772740+0000","last_fullsized":"2024-09-22T03:04:53.772740+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17629184,"num_objects":269,"num_object_clones":0,"num_object_copies":807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":147,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121377+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:46.121377+0000","last_peered":"2024-09-22T03:04:46.121377+0000","last_clean":"2024-09-22T03:04:46.121377+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:46.121377+0000","last_undegraded":"2024-09-22T03:04:46.121377+0000","last_fullsized":"2024-09-22T03:04:46.121377+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'3628","reported_seq":4825,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:53.093461+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:53.093461+0000","last_peered":"2024-09-22T03:04:53.093461+0000","last_clean":"2024-09-22T03:04:53.093461+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:53.093461+0000","last_undegraded":"2024-09-22T03:04:53.093461+0000","last_fullsized":"2024-09-22T03:04:53.093461+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3628,"log_dups_size":0,"ondisk_log_size":3628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18939904,"num_objects":289,"num_object_clones":0,"num_object_copies":867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3629,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":133,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.127168+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:46.127168+0000","last_peered":"2024-09-22T03:04:46.127168+0000","last_clean":"2024-09-22T03:04:46.127168+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:46.127168+0000","last_undegraded":"2024-09-22T03:04:46.127168+0000","last_fullsized":"2024-09-22T03:04:46.127168+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"55'3441","reported_seq":4494,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:49.742830+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:04:49.742830+0000","last_peered":"2024-09-22T03:04:49.742830+0000","last_clean":"2024-09-22T03:04:49.742830+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:04:49.742830+0000","last_undegraded":"2024-09-22T03:04:49.742830+0000","last_fullsized":"2024-09-22T03:04:49.742830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3441,"log_dups_size":0,"ondisk_log_size":3441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":328,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":128,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.120371+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:46.120371+0000","last_peered":"2024-09-22T03:04:46.120371+0000","last_clean":"2024-09-22T03:04:46.120371+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:46.120371+0000","last_undegraded":"2024-09-22T03:04:46.120371+0000","last_fullsized":"2024-09-22T03:04:46.120371+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":270,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.126224+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:46.126224+0000","last_peered":"2024-09-22T03:04:46.126224+0000","last_clean":"2024-09-22T03:04:46.126224+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:46.126224+0000","last_undegraded":"2024-09-22T03:04:46.126224+0000","last_fullsized":"2024-09-22T03:04:46.126224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"55'3736","reported_seq":3879,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:49.838845+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:49.838845+0000","last_peered":"2024-09-22T03:04:49.838845+0000","last_clean":"2024-09-22T03:04:49.838845+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:49.838845+0000","last_undegraded":"2024-09-22T03:04:49.838845+0000","last_fullsized":"2024-09-22T03:04:49.838845+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21037056,"num_objects":321,"num_object_clones":0,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"55'3634","reported_seq":4206,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:53.810527+0000","last_change":"2024-09-22T03:04:43.320486+0000","last_active":"2024-09-22T03:04:53.810527+0000","last_peered":"2024-09-22T03:04:53.810527+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:53.810527+0000","last_undegraded":"2024-09-22T03:04:53.810527+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3634,"log_dups_size":0,"ondisk_log_size":3634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":290,"num_object_clones":0,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3634,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":290}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":101,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.139422+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:46.139422+0000","last_peered":"2024-09-22T03:04:46.139422+0000","last_clean":"2024-09-22T03:04:46.139422+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:46.139422+0000","last_undegraded":"2024-09-22T03:04:46.139422+0000","last_fullsized":"2024-09-22T03:04:46.139422+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"55'3623","reported_seq":3760,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:50.028562+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:50.028562+0000","last_peered":"2024-09-22T03:04:50.028562+0000","last_clean":"2024-09-22T03:04:50.028562+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:50.028562+0000","last_undegraded":"2024-09-22T03:04:50.028562+0000","last_fullsized":"2024-09-22T03:04:50.028562+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":352,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3624,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121224+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:46.121224+0000","last_peered":"2024-09-22T03:04:46.121224+0000","last_clean":"2024-09-22T03:04:46.121224+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:46.121224+0000","last_undegraded":"2024-09-22T03:04:46.121224+0000","last_fullsized":"2024-09-22T03:04:46.121224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"55'3492","reported_seq":4529,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:53.756604+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:04:53.756604+0000","last_peered":"2024-09-22T03:04:53.756604+0000","last_clean":"2024-09-22T03:04:53.756604+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:04:53.756604+0000","last_undegraded":"2024-09-22T03:04:53.756604+0000","last_fullsized":"2024-09-22T03:04:53.756604+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3492,"log_dups_size":0,"ondisk_log_size":3492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19202048,"num_objects":293,"num_object_clones":0,"num_object_copies":879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3493,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":108,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.138950+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:46.138950+0000","last_peered":"2024-09-22T03:04:46.138950+0000","last_clean":"2024-09-22T03:04:46.138950+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:46.138950+0000","last_undegraded":"2024-09-22T03:04:46.138950+0000","last_fullsized":"2024-09-22T03:04:46.138950+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"55'3384","reported_seq":4130,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:53.815239+0000","last_change":"2024-09-22T03:04:44.549017+0000","last_active":"2024-09-22T03:04:53.815239+0000","last_peered":"2024-09-22T03:04:53.815239+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:53.815239+0000","last_undegraded":"2024-09-22T03:04:53.815239+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3385,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":313,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":259}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":110,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.140783+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:46.140783+0000","last_peered":"2024-09-22T03:04:46.140783+0000","last_clean":"2024-09-22T03:04:46.140783+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:46.140783+0000","last_undegraded":"2024-09-22T03:04:46.140783+0000","last_fullsized":"2024-09-22T03:04:46.140783+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":306839576,"num_objects":4683,"num_object_clones":0,"num_object_copies":14049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":56729,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4163,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":62611456,"data_stored":1001586760,"data_compressed":47560696,"data_compressed_allocated":62599168,"data_compressed_original":1001586688,"omap_allocated":0,"internal_metadata":0},"log_size":56715,"ondisk_log_size":56715,"up":48,"acting":44,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74088,"kb_used_data":19416,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94297752,"statfs":{"total":96636764160,"available":96560898048,"internally_reserved":0,"allocated":19881984,"data_stored":310881613,"data_compressed":14753992,"data_compressed_allocated":19419136,"data_compressed_original":310706176,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626482,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74900,"kb_used_data":17284,"kb_used_omap":0,"kb_used_meta":57600,"kb_avail":94296940,"statfs":{"total":96636764160,"available":96560066560,"internally_reserved":0,"allocated":17698816,"data_stored":275623765,"data_compressed":13081687,"data_compressed_allocated":17219584,"data_compressed_original":275431424,"omap_allocated":0,"internal_metadata":58982400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673329,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76932,"kb_used_data":14000,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94294908,"statfs":{"total":96636764160,"available":96557985792,"internally_reserved":0,"allocated":14336000,"data_stored":221818714,"data_compressed":10526735,"data_compressed_allocated":13856768,"data_compressed_original":221626368,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75668,"kb_used_data":13952,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94296172,"statfs":{"total":96636764160,"available":96559280128,"internally_reserved":0,"allocated":14286848,"data_stored":220966765,"data_compressed":10486279,"data_compressed_allocated":13803520,"data_compressed_original":220774400,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13385728,"data_stored":214106136,"data_compressed":10166904,"data_compressed_allocated":13381632,"data_compressed_original":214106112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13418496,"data_stored":214630424,"data_compressed":10191800,"data_compressed_allocated":13414400,"data_compressed_original":214630400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17018880,"data_stored":272302080,"data_compressed":12930360,"data_compressed_allocated":17018880,"data_compressed_original":272302080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18788352,"data_stored":300548120,"data_compressed":14271632,"data_compressed_allocated":18784256,"data_compressed_original":300548096,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:55.718 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:04:55.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:55.863+0000 7f39e7957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:55.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:55.870+0000 7f39e7957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:56.340 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:56.340 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 488/14061 objects degraded (3.471%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":382,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974243,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovering+undersized+remapped","count":3},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":4687,"data_bytes":307429963,"bytes_used":308826112,"bytes_avail":386238230528,"bytes_total":386547056640,"degraded_objects":488,"degraded_total":14061,"degraded_ratio":0.034705924187468887,"recovering_objects_per_sec":79,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":736,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":191},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:04:56.359 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:04:56.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:56.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:56.499+0000 7fea3ad97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:56.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:56.508+0000 7fea3ad97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:56.850 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:04:56.851 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:04:56.868 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-09-22T03:04:56.359200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":289931851,"num_objects":4420,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":57135,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4163,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57082,"ondisk_log_size":57082,"up":75,"acting":71,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":301872,"kb_used_data":60772,"kb_used_omap":0,"kb_used_meta":241024,"kb_avail":377185488,"statfs":{"total":386547056640,"available":386237939712,"internally_reserved":0,"allocated":62230528,"data_stored":965720937,"data_compressed":45830053,"data_compressed_allocated":60325888,"data_compressed_original":964968448,"omap_allocated":0,"internal_metadata":246808576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-133562368,"num_objects":-2038,"num_object_clones":0,"num_object_copies":-6114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.248235"},"pg_stats":[{"pgid":"3.f","version":"55'3582","reported_seq":3742,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:52.252914+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:52.252914+0000","last_peered":"2024-09-22T03:04:52.252914+0000","last_clean":"2024-09-22T03:04:52.252914+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:52.252914+0000","last_undegraded":"2024-09-22T03:04:52.252914+0000","last_fullsized":"2024-09-22T03:04:52.252914+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19595264,"num_objects":299,"num_object_clones":0,"num_object_copies":897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3583,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'3543","reported_seq":3679,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:53.746204+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:53.746204+0000","last_peered":"2024-09-22T03:04:53.746204+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:53.746204+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3543,"log_dups_size":0,"ondisk_log_size":3543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18481152,"num_objects":282,"num_object_clones":0,"num_object_copies":846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3544,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":282}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"55'3585","reported_seq":3728,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:53.734194+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:53.734194+0000","last_peered":"2024-09-22T03:04:53.734194+0000","last_clean":"2024-09-22T03:04:53.734194+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:53.734194+0000","last_undegraded":"2024-09-22T03:04:53.734194+0000","last_fullsized":"2024-09-22T03:04:53.734194+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17629208,"num_objects":270,"num_object_clones":0,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3586,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"55'3411","reported_seq":3951,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:52.334618+0000","last_change":"2024-09-22T03:04:44.918075+0000","last_active":"2024-09-22T03:04:52.334618+0000","last_peered":"2024-09-22T03:04:52.334618+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:52.334618+0000","last_undegraded":"2024-09-22T03:04:52.334618+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18546688,"num_objects":283,"num_object_clones":0,"num_object_copies":849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":283}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"55'3539","reported_seq":4615,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:54.928087+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:04:54.928087+0000","last_peered":"2024-09-22T03:04:54.928087+0000","last_clean":"2024-09-22T03:04:54.928087+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:04:54.928087+0000","last_undegraded":"2024-09-22T03:04:54.928087+0000","last_fullsized":"2024-09-22T03:04:54.928087+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3539,"log_dups_size":0,"ondisk_log_size":3539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17694720,"num_objects":270,"num_object_clones":0,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3540,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"55'3486","reported_seq":4613,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:52.263897+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:52.263897+0000","last_peered":"2024-09-22T03:04:52.263897+0000","last_clean":"2024-09-22T03:04:52.263897+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:52.263897+0000","last_undegraded":"2024-09-22T03:04:52.263897+0000","last_fullsized":"2024-09-22T03:04:52.263897+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3486,"log_dups_size":0,"ondisk_log_size":3486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3487,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"55'3458","reported_seq":4518,"reported_epoch":55,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:52.309987+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:52.309987+0000","last_peered":"2024-09-22T03:04:52.309987+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:52.309987+0000","last_undegraded":"2024-09-22T03:04:52.309987+0000","last_fullsized":"2024-09-22T03:04:52.309987+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17760256,"num_objects":271,"num_object_clones":0,"num_object_copies":813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3459,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":271}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"55'3638","reported_seq":4801,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:53.732957+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:53.732957+0000","last_peered":"2024-09-22T03:04:53.732957+0000","last_clean":"2024-09-22T03:04:53.732957+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:53.732957+0000","last_undegraded":"2024-09-22T03:04:53.732957+0000","last_fullsized":"2024-09-22T03:04:53.732957+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3638,"log_dups_size":0,"ondisk_log_size":3638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15925248,"num_objects":243,"num_object_clones":0,"num_object_copies":729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3639,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":118,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121580+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:46.121580+0000","last_peered":"2024-09-22T03:04:46.121580+0000","last_clean":"2024-09-22T03:04:46.121580+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:46.121580+0000","last_undegraded":"2024-09-22T03:04:46.121580+0000","last_fullsized":"2024-09-22T03:04:46.121580+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"55'3596","reported_seq":3740,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:53.772740+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:53.772740+0000","last_peered":"2024-09-22T03:04:53.772740+0000","last_clean":"2024-09-22T03:04:53.772740+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:53.772740+0000","last_undegraded":"2024-09-22T03:04:53.772740+0000","last_fullsized":"2024-09-22T03:04:53.772740+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17629184,"num_objects":269,"num_object_clones":0,"num_object_copies":807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":147,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121377+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:46.121377+0000","last_peered":"2024-09-22T03:04:46.121377+0000","last_clean":"2024-09-22T03:04:46.121377+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:46.121377+0000","last_undegraded":"2024-09-22T03:04:46.121377+0000","last_fullsized":"2024-09-22T03:04:46.121377+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'3628","reported_seq":4825,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:53.093461+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:53.093461+0000","last_peered":"2024-09-22T03:04:53.093461+0000","last_clean":"2024-09-22T03:04:53.093461+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:53.093461+0000","last_undegraded":"2024-09-22T03:04:53.093461+0000","last_fullsized":"2024-09-22T03:04:53.093461+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3628,"log_dups_size":0,"ondisk_log_size":3628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18939904,"num_objects":289,"num_object_clones":0,"num_object_copies":867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3629,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":133,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.127168+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:46.127168+0000","last_peered":"2024-09-22T03:04:46.127168+0000","last_clean":"2024-09-22T03:04:46.127168+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:46.127168+0000","last_undegraded":"2024-09-22T03:04:46.127168+0000","last_fullsized":"2024-09-22T03:04:46.127168+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"55'3504","reported_seq":4557,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:55.022183+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:04:55.022183+0000","last_peered":"2024-09-22T03:04:55.022183+0000","last_clean":"2024-09-22T03:04:55.022183+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:04:55.022183+0000","last_undegraded":"2024-09-22T03:04:55.022183+0000","last_fullsized":"2024-09-22T03:04:55.022183+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3504,"log_dups_size":0,"ondisk_log_size":3504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17367040,"num_objects":265,"num_object_clones":0,"num_object_copies":795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3505,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":128,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.120371+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:46.120371+0000","last_peered":"2024-09-22T03:04:46.120371+0000","last_clean":"2024-09-22T03:04:46.120371+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:46.120371+0000","last_undegraded":"2024-09-22T03:04:46.120371+0000","last_fullsized":"2024-09-22T03:04:46.120371+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":270,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.126224+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:46.126224+0000","last_peered":"2024-09-22T03:04:46.126224+0000","last_clean":"2024-09-22T03:04:46.126224+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:46.126224+0000","last_undegraded":"2024-09-22T03:04:46.126224+0000","last_fullsized":"2024-09-22T03:04:46.126224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"55'3802","reported_seq":3945,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:54.878466+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:54.878466+0000","last_peered":"2024-09-22T03:04:54.878466+0000","last_clean":"2024-09-22T03:04:54.878466+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:54.878466+0000","last_undegraded":"2024-09-22T03:04:54.878466+0000","last_fullsized":"2024-09-22T03:04:54.878466+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3802,"log_dups_size":0,"ondisk_log_size":3802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":255,"num_object_clones":0,"num_object_copies":765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3803,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"55'3634","reported_seq":4206,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:53.810527+0000","last_change":"2024-09-22T03:04:43.320486+0000","last_active":"2024-09-22T03:04:53.810527+0000","last_peered":"2024-09-22T03:04:53.810527+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:53.810527+0000","last_undegraded":"2024-09-22T03:04:53.810527+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3634,"log_dups_size":0,"ondisk_log_size":3634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":290,"num_object_clones":0,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3634,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":290}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":101,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.139422+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:46.139422+0000","last_peered":"2024-09-22T03:04:46.139422+0000","last_clean":"2024-09-22T03:04:46.139422+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:46.139422+0000","last_undegraded":"2024-09-22T03:04:46.139422+0000","last_fullsized":"2024-09-22T03:04:46.139422+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"55'3700","reported_seq":3837,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:54.962746+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:54.962746+0000","last_peered":"2024-09-22T03:04:54.962746+0000","last_clean":"2024-09-22T03:04:54.962746+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:54.962746+0000","last_undegraded":"2024-09-22T03:04:54.962746+0000","last_fullsized":"2024-09-22T03:04:54.962746+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18022400,"num_objects":275,"num_object_clones":0,"num_object_copies":825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3701,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121224+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:46.121224+0000","last_peered":"2024-09-22T03:04:46.121224+0000","last_clean":"2024-09-22T03:04:46.121224+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:46.121224+0000","last_undegraded":"2024-09-22T03:04:46.121224+0000","last_fullsized":"2024-09-22T03:04:46.121224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"55'3492","reported_seq":4529,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:53.756604+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:04:53.756604+0000","last_peered":"2024-09-22T03:04:53.756604+0000","last_clean":"2024-09-22T03:04:53.756604+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:04:53.756604+0000","last_undegraded":"2024-09-22T03:04:53.756604+0000","last_fullsized":"2024-09-22T03:04:53.756604+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3492,"log_dups_size":0,"ondisk_log_size":3492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19202048,"num_objects":293,"num_object_clones":0,"num_object_copies":879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3493,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":108,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.138950+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:46.138950+0000","last_peered":"2024-09-22T03:04:46.138950+0000","last_clean":"2024-09-22T03:04:46.138950+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:46.138950+0000","last_undegraded":"2024-09-22T03:04:46.138950+0000","last_fullsized":"2024-09-22T03:04:46.138950+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"55'3384","reported_seq":4130,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:53.815239+0000","last_change":"2024-09-22T03:04:44.549017+0000","last_active":"2024-09-22T03:04:53.815239+0000","last_peered":"2024-09-22T03:04:53.815239+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:53.815239+0000","last_undegraded":"2024-09-22T03:04:53.815239+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3384,"log_dups_size":0,"ondisk_log_size":3384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3385,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":313,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":259}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":110,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.140783+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:46.140783+0000","last_peered":"2024-09-22T03:04:46.140783+0000","last_clean":"2024-09-22T03:04:46.140783+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:46.140783+0000","last_undegraded":"2024-09-22T03:04:46.140783+0000","last_fullsized":"2024-09-22T03:04:46.140783+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":289341464,"num_objects":4416,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":56996,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4163,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":58679296,"data_stored":938672200,"data_compressed":44573176,"data_compressed_allocated":58667008,"data_compressed_original":938672128,"omap_allocated":0,"internal_metadata":0},"log_size":56982,"ondisk_log_size":56982,"up":48,"acting":44,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74372,"kb_used_data":15536,"kb_used_omap":0,"kb_used_meta":58816,"kb_avail":94297468,"statfs":{"total":96636764160,"available":96560607232,"internally_reserved":0,"allocated":15908864,"data_stored":247311693,"data_compressed":11735352,"data_compressed_allocated":15446016,"data_compressed_original":247136256,"omap_allocated":0,"internal_metadata":60227584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626482,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74900,"kb_used_data":17284,"kb_used_omap":0,"kb_used_meta":57600,"kb_avail":94296940,"statfs":{"total":96636764160,"available":96560066560,"internally_reserved":0,"allocated":17698816,"data_stored":275623765,"data_compressed":13081687,"data_compressed_allocated":17219584,"data_compressed_original":275431424,"omap_allocated":0,"internal_metadata":58982400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673329,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76932,"kb_used_data":14000,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94294908,"statfs":{"total":96636764160,"available":96557985792,"internally_reserved":0,"allocated":14336000,"data_stored":221818714,"data_compressed":10526735,"data_compressed_allocated":13856768,"data_compressed_original":221626368,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75668,"kb_used_data":13952,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94296172,"statfs":{"total":96636764160,"available":96559280128,"internally_reserved":0,"allocated":14286848,"data_stored":220966765,"data_compressed":10486279,"data_compressed_allocated":13803520,"data_compressed_original":220774400,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13385728,"data_stored":214106136,"data_compressed":10166904,"data_compressed_allocated":13381632,"data_compressed_original":214106112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13418496,"data_stored":214630424,"data_compressed":10191800,"data_compressed_allocated":13414400,"data_compressed_original":214630400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17018880,"data_stored":272302080,"data_compressed":12930360,"data_compressed_allocated":17018880,"data_compressed_original":272302080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14856192,"data_stored":237633560,"data_compressed":11284112,"data_compressed_allocated":14852096,"data_compressed_original":237633536,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:04:57.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:04:58.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:58.084+0000 7fe1c1397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:58.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:58.089+0000 7fe1c1397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:59.870 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:04:59.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:59.978+0000 7fa519d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:04:59.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:04:59.986+0000 7fa519d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:00.188 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:00.202 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-22T03:05:00.145709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222429771,"num_objects":3390,"num_object_clones":0,"num_object_copies":10170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":58165,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58113,"ondisk_log_size":58113,"up":75,"acting":71,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":287872,"kb_used_data":46764,"kb_used_omap":0,"kb_used_meta":241024,"kb_avail":377199488,"statfs":{"total":386547056640,"available":386252275712,"internally_reserved":0,"allocated":47886336,"data_stored":736086174,"data_compressed":34925605,"data_compressed_allocated":45973504,"data_compressed_original":735330304,"omap_allocated":0,"internal_metadata":246808576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-139067392,"num_objects":-2122,"num_object_clones":0,"num_object_copies":-6366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2122,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":750,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.787708"},"pg_stats":[{"pgid":"3.f","version":"55'3645","reported_seq":3805,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:57.255235+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:57.255235+0000","last_peered":"2024-09-22T03:04:57.255235+0000","last_clean":"2024-09-22T03:04:57.255235+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:57.255235+0000","last_undegraded":"2024-09-22T03:04:57.255235+0000","last_fullsized":"2024-09-22T03:04:57.255235+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3645,"log_dups_size":0,"ondisk_log_size":3645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":236,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3646,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'3616","reported_seq":3752,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:58.597040+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:58.597040+0000","last_peered":"2024-09-22T03:04:58.597040+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:58.597040+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3616,"log_dups_size":0,"ondisk_log_size":3616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13697024,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":209}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"55'3651","reported_seq":3794,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:58.727765+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:58.727765+0000","last_peered":"2024-09-22T03:04:58.727765+0000","last_clean":"2024-09-22T03:04:58.727765+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:58.727765+0000","last_undegraded":"2024-09-22T03:04:58.727765+0000","last_fullsized":"2024-09-22T03:04:58.727765+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13303832,"num_objects":204,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3652,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"55'3483","reported_seq":4281,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:57.346113+0000","last_change":"2024-09-22T03:04:44.918075+0000","last_active":"2024-09-22T03:04:57.346113+0000","last_peered":"2024-09-22T03:04:57.346113+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:57.346113+0000","last_undegraded":"2024-09-22T03:04:57.346113+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13828096,"num_objects":211,"num_object_clones":0,"num_object_copies":633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"56'3604","reported_seq":4683,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.655917+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:04:59.655917+0000","last_peered":"2024-09-22T03:04:59.655917+0000","last_clean":"2024-09-22T03:04:59.655917+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:04:59.655917+0000","last_undegraded":"2024-09-22T03:04:59.655917+0000","last_fullsized":"2024-09-22T03:04:59.655917+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3604,"log_dups_size":0,"ondisk_log_size":3604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3605,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"55'3563","reported_seq":4690,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:57.298862+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:57.298862+0000","last_peered":"2024-09-22T03:04:57.298862+0000","last_clean":"2024-09-22T03:04:57.298862+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:57.298862+0000","last_undegraded":"2024-09-22T03:04:57.298862+0000","last_fullsized":"2024-09-22T03:04:57.298862+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3564,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"55'3503","reported_seq":4563,"reported_epoch":55,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:57.346277+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:57.346277+0000","last_peered":"2024-09-22T03:04:57.346277+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:57.346277+0000","last_undegraded":"2024-09-22T03:04:57.346277+0000","last_fullsized":"2024-09-22T03:04:57.346277+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":226}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"55'3689","reported_seq":4852,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:58.569899+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:58.569899+0000","last_peered":"2024-09-22T03:04:58.569899+0000","last_clean":"2024-09-22T03:04:58.569899+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:58.569899+0000","last_undegraded":"2024-09-22T03:04:58.569899+0000","last_fullsized":"2024-09-22T03:04:58.569899+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3689,"log_dups_size":0,"ondisk_log_size":3689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3690,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":121,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.168608+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:59.168608+0000","last_peered":"2024-09-22T03:04:59.168608+0000","last_clean":"2024-09-22T03:04:59.168608+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:59.168608+0000","last_undegraded":"2024-09-22T03:04:59.168608+0000","last_fullsized":"2024-09-22T03:04:59.168608+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"55'3661","reported_seq":3805,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:58.687629+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:58.687629+0000","last_peered":"2024-09-22T03:04:58.687629+0000","last_clean":"2024-09-22T03:04:58.687629+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:58.687629+0000","last_undegraded":"2024-09-22T03:04:58.687629+0000","last_fullsized":"2024-09-22T03:04:58.687629+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13369344,"num_objects":204,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3662,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":147,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121377+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:46.121377+0000","last_peered":"2024-09-22T03:04:46.121377+0000","last_clean":"2024-09-22T03:04:46.121377+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:46.121377+0000","last_undegraded":"2024-09-22T03:04:46.121377+0000","last_fullsized":"2024-09-22T03:04:46.121377+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'3704","reported_seq":4901,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:58.723132+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:58.723132+0000","last_peered":"2024-09-22T03:04:58.723132+0000","last_clean":"2024-09-22T03:04:58.723132+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:58.723132+0000","last_undegraded":"2024-09-22T03:04:58.723132+0000","last_fullsized":"2024-09-22T03:04:58.723132+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13959168,"num_objects":213,"num_object_clones":0,"num_object_copies":639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3705,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":133,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.127168+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:46.127168+0000","last_peered":"2024-09-22T03:04:46.127168+0000","last_clean":"2024-09-22T03:04:46.127168+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:46.127168+0000","last_undegraded":"2024-09-22T03:04:46.127168+0000","last_fullsized":"2024-09-22T03:04:46.127168+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"56'3567","reported_seq":4623,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.387868+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:04:59.387868+0000","last_peered":"2024-09-22T03:04:59.387868+0000","last_clean":"2024-09-22T03:04:59.387868+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:04:59.387868+0000","last_undegraded":"2024-09-22T03:04:59.387868+0000","last_fullsized":"2024-09-22T03:04:59.387868+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3567,"log_dups_size":0,"ondisk_log_size":3567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3568,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":131,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.168263+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:59.168263+0000","last_peered":"2024-09-22T03:04:59.168263+0000","last_clean":"2024-09-22T03:04:59.168263+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:59.168263+0000","last_undegraded":"2024-09-22T03:04:59.168263+0000","last_fullsized":"2024-09-22T03:04:59.168263+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":270,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.126224+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:46.126224+0000","last_peered":"2024-09-22T03:04:46.126224+0000","last_clean":"2024-09-22T03:04:46.126224+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:46.126224+0000","last_undegraded":"2024-09-22T03:04:46.126224+0000","last_fullsized":"2024-09-22T03:04:46.126224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"56'3865","reported_seq":4011,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.611878+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:59.611878+0000","last_peered":"2024-09-22T03:04:59.611878+0000","last_clean":"2024-09-22T03:04:59.611878+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:59.611878+0000","last_undegraded":"2024-09-22T03:04:59.611878+0000","last_fullsized":"2024-09-22T03:04:59.611878+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3865,"log_dups_size":0,"ondisk_log_size":3865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3866,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"55'3704","reported_seq":4466,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:58.834156+0000","last_change":"2024-09-22T03:04:43.320486+0000","last_active":"2024-09-22T03:04:58.834156+0000","last_peered":"2024-09-22T03:04:58.834156+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:58.834156+0000","last_undegraded":"2024-09-22T03:04:58.834156+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3704,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":220}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":101,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.139422+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:46.139422+0000","last_peered":"2024-09-22T03:04:46.139422+0000","last_clean":"2024-09-22T03:04:46.139422+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:46.139422+0000","last_undegraded":"2024-09-22T03:04:46.139422+0000","last_fullsized":"2024-09-22T03:04:46.139422+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"55'3755","reported_seq":3895,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.162922+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:59.162922+0000","last_peered":"2024-09-22T03:04:59.162922+0000","last_clean":"2024-09-22T03:04:59.162922+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:59.162922+0000","last_undegraded":"2024-09-22T03:04:59.162922+0000","last_fullsized":"2024-09-22T03:04:59.162922+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3755,"log_dups_size":0,"ondisk_log_size":3755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3756,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121224+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:46.121224+0000","last_peered":"2024-09-22T03:04:46.121224+0000","last_clean":"2024-09-22T03:04:46.121224+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:46.121224+0000","last_undegraded":"2024-09-22T03:04:46.121224+0000","last_fullsized":"2024-09-22T03:04:46.121224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"55'3561","reported_seq":4598,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:58.262612+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:04:58.262612+0000","last_peered":"2024-09-22T03:04:58.262612+0000","last_clean":"2024-09-22T03:04:58.262612+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:04:58.262612+0000","last_undegraded":"2024-09-22T03:04:58.262612+0000","last_fullsized":"2024-09-22T03:04:58.262612+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14680064,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3562,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":108,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.138950+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:46.138950+0000","last_peered":"2024-09-22T03:04:46.138950+0000","last_clean":"2024-09-22T03:04:46.138950+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:46.138950+0000","last_undegraded":"2024-09-22T03:04:46.138950+0000","last_fullsized":"2024-09-22T03:04:46.138950+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"55'3442","reported_seq":4491,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:58.820941+0000","last_change":"2024-09-22T03:04:44.549017+0000","last_active":"2024-09-22T03:04:58.820941+0000","last_peered":"2024-09-22T03:04:58.820941+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:58.820941+0000","last_undegraded":"2024-09-22T03:04:58.820941+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3442,"log_dups_size":0,"ondisk_log_size":3442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":202}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":110,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.140783+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:46.140783+0000","last_peered":"2024-09-22T03:04:46.140783+0000","last_clean":"2024-09-22T03:04:46.140783+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:46.140783+0000","last_undegraded":"2024-09-22T03:04:46.140783+0000","last_fullsized":"2024-09-22T03:04:46.140783+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":221839384,"num_objects":3386,"num_object_clones":0,"num_object_copies":10158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":58026,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":44625920,"data_stored":713818184,"data_compressed":33895904,"data_compressed_allocated":44613632,"data_compressed_original":713818112,"omap_allocated":0,"internal_metadata":0},"log_size":58013,"ondisk_log_size":58013,"up":48,"acting":44,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70724,"kb_used_data":11892,"kb_used_omap":0,"kb_used_meta":58816,"kb_avail":94301116,"statfs":{"total":96636764160,"available":96564342784,"internally_reserved":0,"allocated":12177408,"data_stored":187480706,"data_compressed":8894096,"data_compressed_allocated":11706368,"data_compressed_original":187301888,"omap_allocated":0,"internal_metadata":60227584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78840,"kb_used_data":12900,"kb_used_omap":0,"kb_used_meta":65920,"kb_avail":94293000,"statfs":{"total":96636764160,"available":96556032000,"internally_reserved":0,"allocated":13209600,"data_stored":203796309,"data_compressed":9670935,"data_compressed_allocated":12730368,"data_compressed_original":203603968,"omap_allocated":0,"internal_metadata":67502080},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673334,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78192,"kb_used_data":11088,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94293648,"statfs":{"total":96636764160,"available":96556695552,"internally_reserved":0,"allocated":11354112,"data_stored":174108506,"data_compressed":8261199,"data_compressed_allocated":10874880,"data_compressed_original":173916160,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888228,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60116,"kb_used_data":10884,"kb_used_omap":0,"kb_used_meta":49216,"kb_avail":94311724,"statfs":{"total":96636764160,"available":96575205376,"internally_reserved":0,"allocated":11145216,"data_stored":170700653,"data_compressed":8099375,"data_compressed_allocated":10661888,"data_compressed_original":170508288,"omap_allocated":0,"internal_metadata":50397184},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10461184,"data_stored":167313432,"data_compressed":7944936,"data_compressed_allocated":10457088,"data_compressed_original":167313408,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10256384,"data_stored":164036632,"data_compressed":7789336,"data_compressed_allocated":10252288,"data_compressed_original":164036608,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12521472,"data_stored":200343552,"data_compressed":9513384,"data_compressed_allocated":12521472,"data_compressed_original":200343552,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11386880,"data_stored":182124568,"data_compressed":8648248,"data_compressed_allocated":11382784,"data_compressed_original":182124544,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:00.204 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:05:00.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:00.332+0000 7f8e11f4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:00.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:00.338+0000 7f8e11f4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:00.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:05:00.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:00.548+0000 7f57a0d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:00.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:00.554+0000 7f57a0d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:00.713 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:00.713 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 488/12489 objects degraded (3.907%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":386,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974243,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovering+undersized+remapped","count":3},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":4163,"data_bytes":273089099,"bytes_used":310407168,"bytes_avail":386236649472,"bytes_total":386547056640,"degraded_objects":488,"degraded_total":12489,"degraded_ratio":0.039074385459204102,"recovering_objects_per_sec":72,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":865,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":191},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:05:00.728 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:05:00.729 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:00.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:00.880+0000 7f090a117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:00.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:00.889+0000 7f090a117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:01.234 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:01.234 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:01.253 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-22T03:05:00.359923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":222429771,"num_objects":3390,"num_object_clones":0,"num_object_copies":10170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":58165,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58113,"ondisk_log_size":58113,"up":75,"acting":71,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":287872,"kb_used_data":46764,"kb_used_omap":0,"kb_used_meta":241024,"kb_avail":377199488,"statfs":{"total":386547056640,"available":386252275712,"internally_reserved":0,"allocated":47886336,"data_stored":736086174,"data_compressed":34925605,"data_compressed_allocated":45973504,"data_compressed_original":735330304,"omap_allocated":0,"internal_metadata":246808576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-139067392,"num_objects":-2122,"num_object_clones":0,"num_object_copies":-6366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2122,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":750,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001569"},"pg_stats":[{"pgid":"3.f","version":"55'3645","reported_seq":3805,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:57.255235+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:04:57.255235+0000","last_peered":"2024-09-22T03:04:57.255235+0000","last_clean":"2024-09-22T03:04:57.255235+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:04:57.255235+0000","last_undegraded":"2024-09-22T03:04:57.255235+0000","last_fullsized":"2024-09-22T03:04:57.255235+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3645,"log_dups_size":0,"ondisk_log_size":3645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":236,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3646,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"55'3616","reported_seq":3752,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:04:58.597040+0000","last_change":"2024-09-22T03:04:05.718386+0000","last_active":"2024-09-22T03:04:58.597040+0000","last_peered":"2024-09-22T03:04:58.597040+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:04:58.597040+0000","last_undegraded":"2024-09-22T03:04:05.717572+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3616,"log_dups_size":0,"ondisk_log_size":3616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13697024,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":209}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"55'3651","reported_seq":3794,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:58.727765+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:04:58.727765+0000","last_peered":"2024-09-22T03:04:58.727765+0000","last_clean":"2024-09-22T03:04:58.727765+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:04:58.727765+0000","last_undegraded":"2024-09-22T03:04:58.727765+0000","last_fullsized":"2024-09-22T03:04:58.727765+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13303832,"num_objects":204,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3652,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"55'3483","reported_seq":4281,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:57.346113+0000","last_change":"2024-09-22T03:04:44.918075+0000","last_active":"2024-09-22T03:04:57.346113+0000","last_peered":"2024-09-22T03:04:57.346113+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:04:57.346113+0000","last_undegraded":"2024-09-22T03:04:57.346113+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13828096,"num_objects":211,"num_object_clones":0,"num_object_copies":633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3483,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":211}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"56'3604","reported_seq":4683,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.655917+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:04:59.655917+0000","last_peered":"2024-09-22T03:04:59.655917+0000","last_clean":"2024-09-22T03:04:59.655917+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:04:59.655917+0000","last_undegraded":"2024-09-22T03:04:59.655917+0000","last_fullsized":"2024-09-22T03:04:59.655917+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3604,"log_dups_size":0,"ondisk_log_size":3604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3605,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"55'3563","reported_seq":4690,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:57.298862+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:04:57.298862+0000","last_peered":"2024-09-22T03:04:57.298862+0000","last_clean":"2024-09-22T03:04:57.298862+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:04:57.298862+0000","last_undegraded":"2024-09-22T03:04:57.298862+0000","last_fullsized":"2024-09-22T03:04:57.298862+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3564,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"55'3503","reported_seq":4563,"reported_epoch":55,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:04:57.346277+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:04:57.346277+0000","last_peered":"2024-09-22T03:04:57.346277+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:04:57.346277+0000","last_undegraded":"2024-09-22T03:04:57.346277+0000","last_fullsized":"2024-09-22T03:04:57.346277+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":226}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"55'3689","reported_seq":4852,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:58.569899+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:04:58.569899+0000","last_peered":"2024-09-22T03:04:58.569899+0000","last_clean":"2024-09-22T03:04:58.569899+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:04:58.569899+0000","last_undegraded":"2024-09-22T03:04:58.569899+0000","last_fullsized":"2024-09-22T03:04:58.569899+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3689,"log_dups_size":0,"ondisk_log_size":3689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3690,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":121,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.168608+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:59.168608+0000","last_peered":"2024-09-22T03:04:59.168608+0000","last_clean":"2024-09-22T03:04:59.168608+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:59.168608+0000","last_undegraded":"2024-09-22T03:04:59.168608+0000","last_fullsized":"2024-09-22T03:04:59.168608+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"55'3661","reported_seq":3805,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:58.687629+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:04:58.687629+0000","last_peered":"2024-09-22T03:04:58.687629+0000","last_clean":"2024-09-22T03:04:58.687629+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:04:58.687629+0000","last_undegraded":"2024-09-22T03:04:58.687629+0000","last_fullsized":"2024-09-22T03:04:58.687629+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13369344,"num_objects":204,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3662,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":147,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121377+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:04:46.121377+0000","last_peered":"2024-09-22T03:04:46.121377+0000","last_clean":"2024-09-22T03:04:46.121377+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:04:46.121377+0000","last_undegraded":"2024-09-22T03:04:46.121377+0000","last_fullsized":"2024-09-22T03:04:46.121377+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"55'3704","reported_seq":4901,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:58.723132+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:04:58.723132+0000","last_peered":"2024-09-22T03:04:58.723132+0000","last_clean":"2024-09-22T03:04:58.723132+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:04:58.723132+0000","last_undegraded":"2024-09-22T03:04:58.723132+0000","last_fullsized":"2024-09-22T03:04:58.723132+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13959168,"num_objects":213,"num_object_clones":0,"num_object_copies":639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3705,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":133,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.127168+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:04:46.127168+0000","last_peered":"2024-09-22T03:04:46.127168+0000","last_clean":"2024-09-22T03:04:46.127168+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:04:46.127168+0000","last_undegraded":"2024-09-22T03:04:46.127168+0000","last_fullsized":"2024-09-22T03:04:46.127168+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"56'3567","reported_seq":4623,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.387868+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:04:59.387868+0000","last_peered":"2024-09-22T03:04:59.387868+0000","last_clean":"2024-09-22T03:04:59.387868+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:04:59.387868+0000","last_undegraded":"2024-09-22T03:04:59.387868+0000","last_fullsized":"2024-09-22T03:04:59.387868+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3567,"log_dups_size":0,"ondisk_log_size":3567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3568,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":131,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.168263+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:59.168263+0000","last_peered":"2024-09-22T03:04:59.168263+0000","last_clean":"2024-09-22T03:04:59.168263+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:59.168263+0000","last_undegraded":"2024-09-22T03:04:59.168263+0000","last_fullsized":"2024-09-22T03:04:59.168263+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":270,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.126224+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:04:46.126224+0000","last_peered":"2024-09-22T03:04:46.126224+0000","last_clean":"2024-09-22T03:04:46.126224+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:04:46.126224+0000","last_undegraded":"2024-09-22T03:04:46.126224+0000","last_fullsized":"2024-09-22T03:04:46.126224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"56'3865","reported_seq":4011,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.611878+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:59.611878+0000","last_peered":"2024-09-22T03:04:59.611878+0000","last_clean":"2024-09-22T03:04:59.611878+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:59.611878+0000","last_undegraded":"2024-09-22T03:04:59.611878+0000","last_fullsized":"2024-09-22T03:04:59.611878+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3865,"log_dups_size":0,"ondisk_log_size":3865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3866,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"55'3704","reported_seq":4466,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:58.834156+0000","last_change":"2024-09-22T03:04:43.320486+0000","last_active":"2024-09-22T03:04:58.834156+0000","last_peered":"2024-09-22T03:04:58.834156+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:04:58.834156+0000","last_undegraded":"2024-09-22T03:04:58.834156+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3704,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":220}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":101,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.139422+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:04:46.139422+0000","last_peered":"2024-09-22T03:04:46.139422+0000","last_clean":"2024-09-22T03:04:46.139422+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:04:46.139422+0000","last_undegraded":"2024-09-22T03:04:46.139422+0000","last_fullsized":"2024-09-22T03:04:46.139422+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"55'3755","reported_seq":3895,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.162922+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:59.162922+0000","last_peered":"2024-09-22T03:04:59.162922+0000","last_clean":"2024-09-22T03:04:59.162922+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:59.162922+0000","last_undegraded":"2024-09-22T03:04:59.162922+0000","last_fullsized":"2024-09-22T03:04:59.162922+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3755,"log_dups_size":0,"ondisk_log_size":3755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3756,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.121224+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:04:46.121224+0000","last_peered":"2024-09-22T03:04:46.121224+0000","last_clean":"2024-09-22T03:04:46.121224+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:04:46.121224+0000","last_undegraded":"2024-09-22T03:04:46.121224+0000","last_fullsized":"2024-09-22T03:04:46.121224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"55'3561","reported_seq":4598,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:58.262612+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:04:58.262612+0000","last_peered":"2024-09-22T03:04:58.262612+0000","last_clean":"2024-09-22T03:04:58.262612+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:04:58.262612+0000","last_undegraded":"2024-09-22T03:04:58.262612+0000","last_fullsized":"2024-09-22T03:04:58.262612+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14680064,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3562,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":108,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.138950+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:04:46.138950+0000","last_peered":"2024-09-22T03:04:46.138950+0000","last_clean":"2024-09-22T03:04:46.138950+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:04:46.138950+0000","last_undegraded":"2024-09-22T03:04:46.138950+0000","last_fullsized":"2024-09-22T03:04:46.138950+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"55'3442","reported_seq":4491,"reported_epoch":55,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:04:58.820941+0000","last_change":"2024-09-22T03:04:44.549017+0000","last_active":"2024-09-22T03:04:58.820941+0000","last_peered":"2024-09-22T03:04:58.820941+0000","last_clean":"2024-09-22T03:02:34.882153+0000","last_became_active":"2024-09-22T03:04:05.574018+0000","last_became_peered":"2024-09-22T03:04:05.574018+0000","last_unstale":"2024-09-22T03:04:58.820941+0000","last_undegraded":"2024-09-22T03:04:58.820941+0000","last_fullsized":"2024-09-22T03:04:05.114504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3442,"log_dups_size":0,"ondisk_log_size":3442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":202}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":110,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-22T03:04:46.140783+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:04:46.140783+0000","last_peered":"2024-09-22T03:04:46.140783+0000","last_clean":"2024-09-22T03:04:46.140783+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:04:46.140783+0000","last_undegraded":"2024-09-22T03:04:46.140783+0000","last_fullsized":"2024-09-22T03:04:46.140783+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":221839384,"num_objects":3386,"num_object_clones":0,"num_object_copies":10158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":58026,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4539,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":44625920,"data_stored":713818184,"data_compressed":33895904,"data_compressed_allocated":44613632,"data_compressed_original":713818112,"omap_allocated":0,"internal_metadata":0},"log_size":58013,"ondisk_log_size":58013,"up":48,"acting":44,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70724,"kb_used_data":11892,"kb_used_omap":0,"kb_used_meta":58816,"kb_avail":94301116,"statfs":{"total":96636764160,"available":96564342784,"internally_reserved":0,"allocated":12177408,"data_stored":187480706,"data_compressed":8894096,"data_compressed_allocated":11706368,"data_compressed_original":187301888,"omap_allocated":0,"internal_metadata":60227584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78840,"kb_used_data":12900,"kb_used_omap":0,"kb_used_meta":65920,"kb_avail":94293000,"statfs":{"total":96636764160,"available":96556032000,"internally_reserved":0,"allocated":13209600,"data_stored":203796309,"data_compressed":9670935,"data_compressed_allocated":12730368,"data_compressed_original":203603968,"omap_allocated":0,"internal_metadata":67502080},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673334,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78192,"kb_used_data":11088,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94293648,"statfs":{"total":96636764160,"available":96556695552,"internally_reserved":0,"allocated":11354112,"data_stored":174108506,"data_compressed":8261199,"data_compressed_allocated":10874880,"data_compressed_original":173916160,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888228,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60116,"kb_used_data":10884,"kb_used_omap":0,"kb_used_meta":49216,"kb_avail":94311724,"statfs":{"total":96636764160,"available":96575205376,"internally_reserved":0,"allocated":11145216,"data_stored":170700653,"data_compressed":8099375,"data_compressed_allocated":10661888,"data_compressed_original":170508288,"omap_allocated":0,"internal_metadata":50397184},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10461184,"data_stored":167313432,"data_compressed":7944936,"data_compressed_allocated":10457088,"data_compressed_original":167313408,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10256384,"data_stored":164036632,"data_compressed":7789336,"data_compressed_allocated":10252288,"data_compressed_original":164036608,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12521472,"data_stored":200343552,"data_compressed":9513384,"data_compressed_allocated":12521472,"data_compressed_original":200343552,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11386880,"data_stored":182124568,"data_compressed":8648248,"data_compressed_allocated":11382784,"data_compressed_original":182124544,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:02.902 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:05:03.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:03.037+0000 7f5db9357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:03.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:03.044+0000 7f5db9357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:04.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:04.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:04.394+0000 7f186d8ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:04.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:04.404+0000 7f186d8ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:04.756 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:04.757 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:04.773 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-22T03:05:04.360623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":175112779,"num_objects":2668,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":58887,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4908,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58834,"ondisk_log_size":58834,"up":75,"acting":72,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":274136,"kb_used_data":37208,"kb_used_omap":0,"kb_used_meta":236864,"kb_avail":377213224,"statfs":{"total":386547056640,"available":386266341376,"internally_reserved":0,"allocated":38100992,"data_stored":578499021,"data_compressed":27441245,"data_compressed_allocated":36122624,"data_compressed_original":577716224,"omap_allocated":0,"internal_metadata":242548736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-63438848,"num_objects":-968,"num_object_clones":0,"num_object_copies":-2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":968,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.218107"},"pg_stats":[{"pgid":"3.f","version":"57'3709","reported_seq":3875,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-22T03:05:02.218555+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:05:02.218555+0000","last_peered":"2024-09-22T03:05:02.218555+0000","last_clean":"2024-09-22T03:05:02.218555+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:05:02.218555+0000","last_undegraded":"2024-09-22T03:05:02.218555+0000","last_fullsized":"2024-09-22T03:05:02.218555+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11272192,"num_objects":172,"num_object_clones":0,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3710,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"58'3684","reported_seq":4087,"reported_epoch":58,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:05:03.798719+0000","last_change":"2024-09-22T03:04:59.208812+0000","last_active":"2024-09-22T03:05:03.798719+0000","last_peered":"2024-09-22T03:05:03.798719+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:05:03.798719+0000","last_undegraded":"2024-09-22T03:05:03.798719+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3684,"log_dups_size":0,"ondisk_log_size":3684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9306112,"num_objects":142,"num_object_clones":0,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3684,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":142}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"58'3712","reported_seq":3864,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.672075+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:05:03.672075+0000","last_peered":"2024-09-22T03:05:03.672075+0000","last_clean":"2024-09-22T03:05:03.672075+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:05:03.672075+0000","last_undegraded":"2024-09-22T03:05:03.672075+0000","last_fullsized":"2024-09-22T03:05:03.672075+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3712,"log_dups_size":0,"ondisk_log_size":3712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9306136,"num_objects":143,"num_object_clones":0,"num_object_copies":429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3713,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"57'3532","reported_seq":4604,"reported_epoch":57,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:05:02.341858+0000","last_change":"2024-09-22T03:04:44.918075+0000","last_active":"2024-09-22T03:05:02.341858+0000","last_peered":"2024-09-22T03:05:02.341858+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:05:02.341858+0000","last_undegraded":"2024-09-22T03:05:02.341858+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3532,"log_dups_size":0,"ondisk_log_size":3532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10616832,"num_objects":162,"num_object_clones":0,"num_object_copies":486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3532,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":162}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"56'3604","reported_seq":4683,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.655917+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:04:59.655917+0000","last_peered":"2024-09-22T03:04:59.655917+0000","last_clean":"2024-09-22T03:04:59.655917+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:04:59.655917+0000","last_undegraded":"2024-09-22T03:04:59.655917+0000","last_fullsized":"2024-09-22T03:04:59.655917+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3604,"log_dups_size":0,"ondisk_log_size":3604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3605,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"57'3629","reported_seq":4762,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-22T03:05:02.210858+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:05:02.210858+0000","last_peered":"2024-09-22T03:05:02.210858+0000","last_clean":"2024-09-22T03:05:02.210858+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:05:02.210858+0000","last_undegraded":"2024-09-22T03:05:02.210858+0000","last_fullsized":"2024-09-22T03:05:02.210858+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3629,"log_dups_size":0,"ondisk_log_size":3629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":160,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3630,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"57'3568","reported_seq":4632,"reported_epoch":57,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:05:02.153957+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:05:02.153957+0000","last_peered":"2024-09-22T03:05:02.153957+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:05:02.153957+0000","last_undegraded":"2024-09-22T03:05:02.153957+0000","last_fullsized":"2024-09-22T03:05:02.153957+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3568,"log_dups_size":0,"ondisk_log_size":3568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10551296,"num_objects":161,"num_object_clones":0,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3569,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":161}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"58'3744","reported_seq":4916,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.676725+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:05:03.676725+0000","last_peered":"2024-09-22T03:05:03.676725+0000","last_clean":"2024-09-22T03:05:03.676725+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:05:03.676725+0000","last_undegraded":"2024-09-22T03:05:03.676725+0000","last_fullsized":"2024-09-22T03:05:03.676725+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3744,"log_dups_size":0,"ondisk_log_size":3744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":137,"num_object_clones":0,"num_object_copies":411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3745,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":121,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.168608+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:59.168608+0000","last_peered":"2024-09-22T03:04:59.168608+0000","last_clean":"2024-09-22T03:04:59.168608+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:59.168608+0000","last_undegraded":"2024-09-22T03:04:59.168608+0000","last_fullsized":"2024-09-22T03:04:59.168608+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"58'3716","reported_seq":3869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.827929+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:05:03.827929+0000","last_peered":"2024-09-22T03:05:03.827929+0000","last_clean":"2024-09-22T03:05:03.827929+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:05:03.827929+0000","last_undegraded":"2024-09-22T03:05:03.827929+0000","last_fullsized":"2024-09-22T03:05:03.827929+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3716,"log_dups_size":0,"ondisk_log_size":3716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764864,"num_objects":149,"num_object_clones":0,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":153,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-22T03:05:00.157696+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:00.157696+0000","last_peered":"2024-09-22T03:05:00.157696+0000","last_clean":"2024-09-22T03:05:00.157696+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:00.157696+0000","last_undegraded":"2024-09-22T03:05:00.157696+0000","last_fullsized":"2024-09-22T03:05:00.157696+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"58'3753","reported_seq":4959,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.802202+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:05:03.802202+0000","last_peered":"2024-09-22T03:05:03.802202+0000","last_clean":"2024-09-22T03:05:03.802202+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:05:03.802202+0000","last_undegraded":"2024-09-22T03:05:03.802202+0000","last_fullsized":"2024-09-22T03:05:03.802202+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10747904,"num_objects":164,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3754,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":142,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.198434+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:03.198434+0000","last_peered":"2024-09-22T03:05:03.198434+0000","last_clean":"2024-09-22T03:05:03.198434+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:03.198434+0000","last_undegraded":"2024-09-22T03:05:03.198434+0000","last_fullsized":"2024-09-22T03:05:03.198434+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"56'3567","reported_seq":4623,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.387868+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:04:59.387868+0000","last_peered":"2024-09-22T03:04:59.387868+0000","last_clean":"2024-09-22T03:04:59.387868+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:04:59.387868+0000","last_undegraded":"2024-09-22T03:04:59.387868+0000","last_fullsized":"2024-09-22T03:04:59.387868+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3567,"log_dups_size":0,"ondisk_log_size":3567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3568,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":131,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.168263+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:59.168263+0000","last_peered":"2024-09-22T03:04:59.168263+0000","last_clean":"2024-09-22T03:04:59.168263+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:59.168263+0000","last_undegraded":"2024-09-22T03:04:59.168263+0000","last_fullsized":"2024-09-22T03:04:59.168263+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":276,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-22T03:05:00.160475+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:00.160475+0000","last_peered":"2024-09-22T03:05:00.160475+0000","last_clean":"2024-09-22T03:05:00.160475+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:00.160475+0000","last_undegraded":"2024-09-22T03:05:00.160475+0000","last_fullsized":"2024-09-22T03:05:00.160475+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"56'3865","reported_seq":4011,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.611878+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:59.611878+0000","last_peered":"2024-09-22T03:04:59.611878+0000","last_clean":"2024-09-22T03:04:59.611878+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:59.611878+0000","last_undegraded":"2024-09-22T03:04:59.611878+0000","last_fullsized":"2024-09-22T03:04:59.611878+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3865,"log_dups_size":0,"ondisk_log_size":3865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3866,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"58'3770","reported_seq":4729,"reported_epoch":58,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:05:03.753713+0000","last_change":"2024-09-22T03:04:43.320486+0000","last_active":"2024-09-22T03:05:03.753713+0000","last_peered":"2024-09-22T03:05:03.753713+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:05:03.753713+0000","last_undegraded":"2024-09-22T03:05:03.753713+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3770,"log_dups_size":0,"ondisk_log_size":3770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10027008,"num_objects":153,"num_object_clones":0,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3771,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":153}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":110,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.191039+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:03.191039+0000","last_peered":"2024-09-22T03:05:03.191039+0000","last_clean":"2024-09-22T03:05:03.191039+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:03.191039+0000","last_undegraded":"2024-09-22T03:05:03.191039+0000","last_fullsized":"2024-09-22T03:05:03.191039+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"55'3755","reported_seq":3895,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.162922+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:59.162922+0000","last_peered":"2024-09-22T03:04:59.162922+0000","last_clean":"2024-09-22T03:04:59.162922+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:59.162922+0000","last_undegraded":"2024-09-22T03:04:59.162922+0000","last_fullsized":"2024-09-22T03:04:59.162922+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3755,"log_dups_size":0,"ondisk_log_size":3755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3756,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.188196+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:03.188196+0000","last_peered":"2024-09-22T03:05:03.188196+0000","last_clean":"2024-09-22T03:05:03.188196+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:03.188196+0000","last_undegraded":"2024-09-22T03:05:03.188196+0000","last_fullsized":"2024-09-22T03:05:03.188196+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"58'3634","reported_seq":4680,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.772547+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:05:03.772547+0000","last_peered":"2024-09-22T03:05:03.772547+0000","last_clean":"2024-09-22T03:05:03.772547+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:05:03.772547+0000","last_undegraded":"2024-09-22T03:05:03.772547+0000","last_fullsized":"2024-09-22T03:05:03.772547+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3634,"log_dups_size":0,"ondisk_log_size":3634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9895936,"num_objects":151,"num_object_clones":0,"num_object_copies":453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":117,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.193097+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:03.193097+0000","last_peered":"2024-09-22T03:05:03.193097+0000","last_clean":"2024-09-22T03:05:03.193097+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:03.193097+0000","last_undegraded":"2024-09-22T03:05:03.193097+0000","last_fullsized":"2024-09-22T03:05:03.193097+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"58'3492","reported_seq":4573,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.555599+0000","last_change":"2024-09-22T03:05:00.209116+0000","last_active":"2024-09-22T03:05:03.555599+0000","last_peered":"2024-09-22T03:05:03.555599+0000","last_clean":"2024-09-22T03:05:03.555599+0000","last_became_active":"2024-09-22T03:05:00.203012+0000","last_became_peered":"2024-09-22T03:05:00.203012+0000","last_unstale":"2024-09-22T03:05:03.555599+0000","last_undegraded":"2024-09-22T03:05:03.555599+0000","last_fullsized":"2024-09-22T03:05:03.555599+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3492,"log_dups_size":0,"ondisk_log_size":3492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9895936,"num_objects":151,"num_object_clones":0,"num_object_copies":453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3493,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":119,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.193438+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:03.193438+0000","last_peered":"2024-09-22T03:05:03.193438+0000","last_clean":"2024-09-22T03:05:03.193438+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:03.193438+0000","last_undegraded":"2024-09-22T03:05:03.193438+0000","last_fullsized":"2024-09-22T03:05:03.193438+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":174522392,"num_objects":2664,"num_object_clones":0,"num_object_copies":7992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":58748,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4908,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34861056,"data_stored":557580360,"data_compressed":26476896,"data_compressed_allocated":34848768,"data_compressed_original":557580288,"omap_allocated":0,"internal_metadata":0},"log_size":58734,"ondisk_log_size":58734,"up":48,"acting":45,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70724,"kb_used_data":11892,"kb_used_omap":0,"kb_used_meta":58816,"kb_avail":94301116,"statfs":{"total":96636764160,"available":96564342784,"internally_reserved":0,"allocated":12177408,"data_stored":187480706,"data_compressed":8894096,"data_compressed_allocated":11706368,"data_compressed_original":187301888,"omap_allocated":0,"internal_metadata":60227584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626492,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62392,"kb_used_data":8936,"kb_used_omap":0,"kb_used_meta":53440,"kb_avail":94309448,"statfs":{"total":96636764160,"available":96572874752,"internally_reserved":0,"allocated":9150464,"data_stored":138467010,"data_compressed":6568271,"data_compressed_allocated":8646656,"data_compressed_original":138264576,"omap_allocated":0,"internal_metadata":54722560},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79660,"kb_used_data":8412,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94292180,"statfs":{"total":96636764160,"available":96555192320,"internally_reserved":0,"allocated":8613888,"data_stored":130009519,"data_compressed":6166823,"data_compressed_allocated":8118272,"data_compressed_original":129810432,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888233,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":61360,"kb_used_data":7968,"kb_used_omap":0,"kb_used_meta":53376,"kb_avail":94310480,"statfs":{"total":96636764160,"available":96573931520,"internally_reserved":0,"allocated":8159232,"data_stored":122541786,"data_compressed":5812055,"data_compressed_allocated":7651328,"data_compressed_original":122339328,"omap_allocated":0,"internal_metadata":54657024},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7692288,"data_stored":123011096,"data_compressed":5841224,"data_compressed_allocated":7688192,"data_compressed_original":123011072,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7299072,"data_stored":116719640,"data_compressed":5542472,"data_compressed_allocated":7294976,"data_compressed_original":116719616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":135725056,"data_compressed":6444952,"data_compressed_allocated":8482816,"data_compressed_original":135725056,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11386880,"data_stored":182124568,"data_compressed":8648248,"data_compressed_allocated":11382784,"data_compressed_original":182124544,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:04.775 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:05:04.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:04.921+0000 7ff70c397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:04.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:04.928+0000 7ff70c397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:05.420 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":391,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974243,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovering+undersized+remapped","count":3},{"state_name":"active+recovery_wait","count":1}],"num_pgs":25,"num_pools":3,"num_objects":2668,"data_bytes":175112779,"bytes_used":280715264,"bytes_avail":386266341376,"bytes_total":386547056640,"recovering_objects_per_sec":70,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":369,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":185},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:05:05.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:05:05.439 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:05:05.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:05.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:05.567+0000 7f2092ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:05.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:05.582+0000 7f2092ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:05.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:05.620+0000 7f786af97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:05.631+0000 7f786af97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:05.938 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:05.938 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:05.954 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-22T03:05:04.360623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":175112779,"num_objects":2668,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":58887,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4908,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58834,"ondisk_log_size":58834,"up":75,"acting":72,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":274136,"kb_used_data":37208,"kb_used_omap":0,"kb_used_meta":236864,"kb_avail":377213224,"statfs":{"total":386547056640,"available":386266341376,"internally_reserved":0,"allocated":38100992,"data_stored":578499021,"data_compressed":27441245,"data_compressed_allocated":36122624,"data_compressed_original":577716224,"omap_allocated":0,"internal_metadata":242548736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-63438848,"num_objects":-968,"num_object_clones":0,"num_object_copies":-2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":968,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.218107"},"pg_stats":[{"pgid":"3.f","version":"57'3709","reported_seq":3875,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-22T03:05:02.218555+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:05:02.218555+0000","last_peered":"2024-09-22T03:05:02.218555+0000","last_clean":"2024-09-22T03:05:02.218555+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:05:02.218555+0000","last_undegraded":"2024-09-22T03:05:02.218555+0000","last_fullsized":"2024-09-22T03:05:02.218555+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11272192,"num_objects":172,"num_object_clones":0,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3710,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"58'3684","reported_seq":4087,"reported_epoch":58,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:05:03.798719+0000","last_change":"2024-09-22T03:04:59.208812+0000","last_active":"2024-09-22T03:05:03.798719+0000","last_peered":"2024-09-22T03:05:03.798719+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:05:03.798719+0000","last_undegraded":"2024-09-22T03:05:03.798719+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3684,"log_dups_size":0,"ondisk_log_size":3684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9306112,"num_objects":142,"num_object_clones":0,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3684,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":142}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"58'3712","reported_seq":3864,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.672075+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:05:03.672075+0000","last_peered":"2024-09-22T03:05:03.672075+0000","last_clean":"2024-09-22T03:05:03.672075+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:05:03.672075+0000","last_undegraded":"2024-09-22T03:05:03.672075+0000","last_fullsized":"2024-09-22T03:05:03.672075+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3712,"log_dups_size":0,"ondisk_log_size":3712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9306136,"num_objects":143,"num_object_clones":0,"num_object_copies":429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3713,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"57'3532","reported_seq":4604,"reported_epoch":57,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:05:02.341858+0000","last_change":"2024-09-22T03:04:44.918075+0000","last_active":"2024-09-22T03:05:02.341858+0000","last_peered":"2024-09-22T03:05:02.341858+0000","last_clean":"2024-09-22T03:02:34.909112+0000","last_became_active":"2024-09-22T03:04:05.642731+0000","last_became_peered":"2024-09-22T03:04:05.642731+0000","last_unstale":"2024-09-22T03:05:02.341858+0000","last_undegraded":"2024-09-22T03:05:02.341858+0000","last_fullsized":"2024-09-22T03:04:05.191088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3532,"log_dups_size":0,"ondisk_log_size":3532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10616832,"num_objects":162,"num_object_clones":0,"num_object_copies":486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3532,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":162}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"56'3604","reported_seq":4683,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.655917+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:04:59.655917+0000","last_peered":"2024-09-22T03:04:59.655917+0000","last_clean":"2024-09-22T03:04:59.655917+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:04:59.655917+0000","last_undegraded":"2024-09-22T03:04:59.655917+0000","last_fullsized":"2024-09-22T03:04:59.655917+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3604,"log_dups_size":0,"ondisk_log_size":3604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3605,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"57'3629","reported_seq":4762,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-22T03:05:02.210858+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:05:02.210858+0000","last_peered":"2024-09-22T03:05:02.210858+0000","last_clean":"2024-09-22T03:05:02.210858+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:05:02.210858+0000","last_undegraded":"2024-09-22T03:05:02.210858+0000","last_fullsized":"2024-09-22T03:05:02.210858+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3629,"log_dups_size":0,"ondisk_log_size":3629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":160,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3630,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"57'3568","reported_seq":4632,"reported_epoch":57,"state":"active+recovery_wait","last_fresh":"2024-09-22T03:05:02.153957+0000","last_change":"2024-09-22T03:04:27.279725+0000","last_active":"2024-09-22T03:05:02.153957+0000","last_peered":"2024-09-22T03:05:02.153957+0000","last_clean":"2024-09-22T03:03:02.498852+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:05:02.153957+0000","last_undegraded":"2024-09-22T03:05:02.153957+0000","last_fullsized":"2024-09-22T03:05:02.153957+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3568,"log_dups_size":0,"ondisk_log_size":3568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10551296,"num_objects":161,"num_object_clones":0,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3569,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":161}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"58'3744","reported_seq":4916,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.676725+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:05:03.676725+0000","last_peered":"2024-09-22T03:05:03.676725+0000","last_clean":"2024-09-22T03:05:03.676725+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:05:03.676725+0000","last_undegraded":"2024-09-22T03:05:03.676725+0000","last_fullsized":"2024-09-22T03:05:03.676725+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3744,"log_dups_size":0,"ondisk_log_size":3744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8978432,"num_objects":137,"num_object_clones":0,"num_object_copies":411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3745,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":121,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.168608+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:04:59.168608+0000","last_peered":"2024-09-22T03:04:59.168608+0000","last_clean":"2024-09-22T03:04:59.168608+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:04:59.168608+0000","last_undegraded":"2024-09-22T03:04:59.168608+0000","last_fullsized":"2024-09-22T03:04:59.168608+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"58'3716","reported_seq":3869,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.827929+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:05:03.827929+0000","last_peered":"2024-09-22T03:05:03.827929+0000","last_clean":"2024-09-22T03:05:03.827929+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:05:03.827929+0000","last_undegraded":"2024-09-22T03:05:03.827929+0000","last_fullsized":"2024-09-22T03:05:03.827929+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3716,"log_dups_size":0,"ondisk_log_size":3716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764864,"num_objects":149,"num_object_clones":0,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":153,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-22T03:05:00.157696+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:00.157696+0000","last_peered":"2024-09-22T03:05:00.157696+0000","last_clean":"2024-09-22T03:05:00.157696+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:00.157696+0000","last_undegraded":"2024-09-22T03:05:00.157696+0000","last_fullsized":"2024-09-22T03:05:00.157696+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"58'3753","reported_seq":4959,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.802202+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:05:03.802202+0000","last_peered":"2024-09-22T03:05:03.802202+0000","last_clean":"2024-09-22T03:05:03.802202+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:05:03.802202+0000","last_undegraded":"2024-09-22T03:05:03.802202+0000","last_fullsized":"2024-09-22T03:05:03.802202+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10747904,"num_objects":164,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3754,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":142,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.198434+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:03.198434+0000","last_peered":"2024-09-22T03:05:03.198434+0000","last_clean":"2024-09-22T03:05:03.198434+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:03.198434+0000","last_undegraded":"2024-09-22T03:05:03.198434+0000","last_fullsized":"2024-09-22T03:05:03.198434+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"56'3567","reported_seq":4623,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.387868+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:04:59.387868+0000","last_peered":"2024-09-22T03:04:59.387868+0000","last_clean":"2024-09-22T03:04:59.387868+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:04:59.387868+0000","last_undegraded":"2024-09-22T03:04:59.387868+0000","last_fullsized":"2024-09-22T03:04:59.387868+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3567,"log_dups_size":0,"ondisk_log_size":3567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3568,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":131,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.168263+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:04:59.168263+0000","last_peered":"2024-09-22T03:04:59.168263+0000","last_clean":"2024-09-22T03:04:59.168263+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:04:59.168263+0000","last_undegraded":"2024-09-22T03:04:59.168263+0000","last_fullsized":"2024-09-22T03:04:59.168263+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":276,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-22T03:05:00.160475+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:00.160475+0000","last_peered":"2024-09-22T03:05:00.160475+0000","last_clean":"2024-09-22T03:05:00.160475+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:00.160475+0000","last_undegraded":"2024-09-22T03:05:00.160475+0000","last_fullsized":"2024-09-22T03:05:00.160475+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"56'3865","reported_seq":4011,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.611878+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:04:59.611878+0000","last_peered":"2024-09-22T03:04:59.611878+0000","last_clean":"2024-09-22T03:04:59.611878+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:04:59.611878+0000","last_undegraded":"2024-09-22T03:04:59.611878+0000","last_fullsized":"2024-09-22T03:04:59.611878+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3865,"log_dups_size":0,"ondisk_log_size":3865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3866,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"58'3770","reported_seq":4729,"reported_epoch":58,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:05:03.753713+0000","last_change":"2024-09-22T03:04:43.320486+0000","last_active":"2024-09-22T03:05:03.753713+0000","last_peered":"2024-09-22T03:05:03.753713+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:05:03.753713+0000","last_undegraded":"2024-09-22T03:05:03.753713+0000","last_fullsized":"2024-09-22T03:04:05.161088+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3770,"log_dups_size":0,"ondisk_log_size":3770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10027008,"num_objects":153,"num_object_clones":0,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3771,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":153}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":110,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.191039+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:03.191039+0000","last_peered":"2024-09-22T03:05:03.191039+0000","last_clean":"2024-09-22T03:05:03.191039+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:03.191039+0000","last_undegraded":"2024-09-22T03:05:03.191039+0000","last_fullsized":"2024-09-22T03:05:03.191039+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"55'3755","reported_seq":3895,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-22T03:04:59.162922+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:04:59.162922+0000","last_peered":"2024-09-22T03:04:59.162922+0000","last_clean":"2024-09-22T03:04:59.162922+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:04:59.162922+0000","last_undegraded":"2024-09-22T03:04:59.162922+0000","last_fullsized":"2024-09-22T03:04:59.162922+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3755,"log_dups_size":0,"ondisk_log_size":3755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3756,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.188196+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:03.188196+0000","last_peered":"2024-09-22T03:05:03.188196+0000","last_clean":"2024-09-22T03:05:03.188196+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:03.188196+0000","last_undegraded":"2024-09-22T03:05:03.188196+0000","last_fullsized":"2024-09-22T03:05:03.188196+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"58'3634","reported_seq":4680,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.772547+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:05:03.772547+0000","last_peered":"2024-09-22T03:05:03.772547+0000","last_clean":"2024-09-22T03:05:03.772547+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:05:03.772547+0000","last_undegraded":"2024-09-22T03:05:03.772547+0000","last_fullsized":"2024-09-22T03:05:03.772547+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3634,"log_dups_size":0,"ondisk_log_size":3634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9895936,"num_objects":151,"num_object_clones":0,"num_object_copies":453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":117,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.193097+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:03.193097+0000","last_peered":"2024-09-22T03:05:03.193097+0000","last_clean":"2024-09-22T03:05:03.193097+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:03.193097+0000","last_undegraded":"2024-09-22T03:05:03.193097+0000","last_fullsized":"2024-09-22T03:05:03.193097+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"58'3492","reported_seq":4573,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.555599+0000","last_change":"2024-09-22T03:05:00.209116+0000","last_active":"2024-09-22T03:05:03.555599+0000","last_peered":"2024-09-22T03:05:03.555599+0000","last_clean":"2024-09-22T03:05:03.555599+0000","last_became_active":"2024-09-22T03:05:00.203012+0000","last_became_peered":"2024-09-22T03:05:00.203012+0000","last_unstale":"2024-09-22T03:05:03.555599+0000","last_undegraded":"2024-09-22T03:05:03.555599+0000","last_fullsized":"2024-09-22T03:05:03.555599+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3492,"log_dups_size":0,"ondisk_log_size":3492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9895936,"num_objects":151,"num_object_clones":0,"num_object_copies":453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3493,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":119,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-22T03:05:03.193438+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:03.193438+0000","last_peered":"2024-09-22T03:05:03.193438+0000","last_clean":"2024-09-22T03:05:03.193438+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:03.193438+0000","last_undegraded":"2024-09-22T03:05:03.193438+0000","last_fullsized":"2024-09-22T03:05:03.193438+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":174522392,"num_objects":2664,"num_object_clones":0,"num_object_copies":7992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":58748,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4908,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34861056,"data_stored":557580360,"data_compressed":26476896,"data_compressed_allocated":34848768,"data_compressed_original":557580288,"omap_allocated":0,"internal_metadata":0},"log_size":58734,"ondisk_log_size":58734,"up":48,"acting":45,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70724,"kb_used_data":11892,"kb_used_omap":0,"kb_used_meta":58816,"kb_avail":94301116,"statfs":{"total":96636764160,"available":96564342784,"internally_reserved":0,"allocated":12177408,"data_stored":187480706,"data_compressed":8894096,"data_compressed_allocated":11706368,"data_compressed_original":187301888,"omap_allocated":0,"internal_metadata":60227584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626492,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62392,"kb_used_data":8936,"kb_used_omap":0,"kb_used_meta":53440,"kb_avail":94309448,"statfs":{"total":96636764160,"available":96572874752,"internally_reserved":0,"allocated":9150464,"data_stored":138467010,"data_compressed":6568271,"data_compressed_allocated":8646656,"data_compressed_original":138264576,"omap_allocated":0,"internal_metadata":54722560},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79660,"kb_used_data":8412,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94292180,"statfs":{"total":96636764160,"available":96555192320,"internally_reserved":0,"allocated":8613888,"data_stored":130009519,"data_compressed":6166823,"data_compressed_allocated":8118272,"data_compressed_original":129810432,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888233,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":61360,"kb_used_data":7968,"kb_used_omap":0,"kb_used_meta":53376,"kb_avail":94310480,"statfs":{"total":96636764160,"available":96573931520,"internally_reserved":0,"allocated":8159232,"data_stored":122541786,"data_compressed":5812055,"data_compressed_allocated":7651328,"data_compressed_original":122339328,"omap_allocated":0,"internal_metadata":54657024},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7692288,"data_stored":123011096,"data_compressed":5841224,"data_compressed_allocated":7688192,"data_compressed_original":123011072,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7299072,"data_stored":116719640,"data_compressed":5542472,"data_compressed_allocated":7294976,"data_compressed_original":116719616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":135725056,"data_compressed":6444952,"data_compressed_allocated":8482816,"data_compressed_original":135725056,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11386880,"data_stored":182124568,"data_compressed":8648248,"data_compressed_allocated":11382784,"data_compressed_original":182124544,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:08.451 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:05:08.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:08.589+0000 7f3b3e8fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:08.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:08.596+0000 7f3b3e8fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:08.957 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:09.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:09.058+0000 7f6c2b0a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:09.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:09.065+0000 7f6c2b0a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:09.260 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:09.261 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:09.272 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-22T03:05:09.167556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":111149643,"num_objects":1692,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":59863,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5094,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59810,"ondisk_log_size":59810,"up":75,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":274220,"kb_used_data":24812,"kb_used_omap":0,"kb_used_meta":249344,"kb_avail":377213140,"statfs":{"total":386547056640,"available":386266255360,"internally_reserved":0,"allocated":25407488,"data_stored":374253343,"data_compressed":17741141,"data_compressed_allocated":23355392,"data_compressed_original":373440512,"omap_allocated":0,"internal_metadata":255328256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-95289344,"num_objects":-1454,"num_object_clones":0,"num_object_copies":-4362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1454,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.015988"},"pg_stats":[{"pgid":"3.f","version":"59'3767","reported_seq":3939,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.096014+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:05:07.096014+0000","last_peered":"2024-09-22T03:05:07.096014+0000","last_clean":"2024-09-22T03:05:07.096014+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:05:07.096014+0000","last_undegraded":"2024-09-22T03:05:07.096014+0000","last_fullsized":"2024-09-22T03:05:07.096014+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7471104,"num_objects":114,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3768,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"60'3738","reported_seq":4374,"reported_epoch":60,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:05:08.820299+0000","last_change":"2024-09-22T03:04:59.208812+0000","last_active":"2024-09-22T03:05:08.820299+0000","last_peered":"2024-09-22T03:05:08.820299+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:05:08.820299+0000","last_undegraded":"2024-09-22T03:05:08.820299+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3738,"log_dups_size":0,"ondisk_log_size":3738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5767168,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3738,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":88}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"60'3767","reported_seq":3925,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.547949+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:05:08.547949+0000","last_peered":"2024-09-22T03:05:08.547949+0000","last_clean":"2024-09-22T03:05:08.547949+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:05:08.547949+0000","last_undegraded":"2024-09-22T03:05:08.547949+0000","last_fullsized":"2024-09-22T03:05:08.547949+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701656,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3768,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"59'3582","reported_seq":4677,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.085692+0000","last_change":"2024-09-22T03:05:04.213909+0000","last_active":"2024-09-22T03:05:07.085692+0000","last_peered":"2024-09-22T03:05:07.085692+0000","last_clean":"2024-09-22T03:05:07.085692+0000","last_became_active":"2024-09-22T03:05:04.207449+0000","last_became_peered":"2024-09-22T03:05:04.207449+0000","last_unstale":"2024-09-22T03:05:07.085692+0000","last_undegraded":"2024-09-22T03:05:07.085692+0000","last_fullsized":"2024-09-22T03:05:07.085692+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3583,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"59'3673","reported_seq":4761,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:05.008368+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:05:05.008368+0000","last_peered":"2024-09-22T03:05:05.008368+0000","last_clean":"2024-09-22T03:05:05.008368+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:05:05.008368+0000","last_undegraded":"2024-09-22T03:05:05.008368+0000","last_fullsized":"2024-09-22T03:05:05.008368+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3673,"log_dups_size":0,"ondisk_log_size":3673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8912896,"num_objects":136,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3674,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"59'3690","reported_seq":4829,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.262590+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:05:07.262590+0000","last_peered":"2024-09-22T03:05:07.262590+0000","last_clean":"2024-09-22T03:05:07.262590+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:05:07.262590+0000","last_undegraded":"2024-09-22T03:05:07.262590+0000","last_fullsized":"2024-09-22T03:05:07.262590+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6488064,"num_objects":99,"num_object_clones":0,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3691,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"59'3627","reported_seq":4700,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.017529+0000","last_change":"2024-09-22T03:05:02.795956+0000","last_active":"2024-09-22T03:05:07.017529+0000","last_peered":"2024-09-22T03:05:07.017529+0000","last_clean":"2024-09-22T03:05:07.017529+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:05:07.017529+0000","last_undegraded":"2024-09-22T03:05:07.017529+0000","last_fullsized":"2024-09-22T03:05:07.017529+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6684672,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3628,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"60'3808","reported_seq":4986,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.599559+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:05:08.599559+0000","last_peered":"2024-09-22T03:05:08.599559+0000","last_clean":"2024-09-22T03:05:08.599559+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:05:08.599559+0000","last_undegraded":"2024-09-22T03:05:08.599559+0000","last_fullsized":"2024-09-22T03:05:08.599559+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3808,"log_dups_size":0,"ondisk_log_size":3808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4784128,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":130,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:04.164410+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:04.164410+0000","last_peered":"2024-09-22T03:05:04.164410+0000","last_clean":"2024-09-22T03:05:04.164410+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:04.164410+0000","last_undegraded":"2024-09-22T03:05:04.164410+0000","last_fullsized":"2024-09-22T03:05:04.164410+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"60'3778","reported_seq":3937,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.455212+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:05:08.455212+0000","last_peered":"2024-09-22T03:05:08.455212+0000","last_clean":"2024-09-22T03:05:08.455212+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:05:08.455212+0000","last_undegraded":"2024-09-22T03:05:08.455212+0000","last_fullsized":"2024-09-22T03:05:08.455212+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3779,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":159,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:04.179196+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:04.179196+0000","last_peered":"2024-09-22T03:05:04.179196+0000","last_clean":"2024-09-22T03:05:04.179196+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:04.179196+0000","last_undegraded":"2024-09-22T03:05:04.179196+0000","last_fullsized":"2024-09-22T03:05:04.179196+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"59'3809","reported_seq":5021,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.219637+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:05:08.219637+0000","last_peered":"2024-09-22T03:05:08.219637+0000","last_clean":"2024-09-22T03:05:08.219637+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:05:08.219637+0000","last_undegraded":"2024-09-22T03:05:08.219637+0000","last_fullsized":"2024-09-22T03:05:08.219637+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7077888,"num_objects":108,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3810,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":148,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.218821+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:08.218821+0000","last_peered":"2024-09-22T03:05:08.218821+0000","last_clean":"2024-09-22T03:05:08.218821+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:08.218821+0000","last_undegraded":"2024-09-22T03:05:08.218821+0000","last_fullsized":"2024-09-22T03:05:08.218821+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"59'3644","reported_seq":4709,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:04.959796+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:05:04.959796+0000","last_peered":"2024-09-22T03:05:04.959796+0000","last_clean":"2024-09-22T03:05:04.959796+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:05:04.959796+0000","last_undegraded":"2024-09-22T03:05:04.959796+0000","last_fullsized":"2024-09-22T03:05:04.959796+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3644,"log_dups_size":0,"ondisk_log_size":3644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8192000,"num_objects":125,"num_object_clones":0,"num_object_copies":375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3645,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":140,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:04.164793+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:04.164793+0000","last_peered":"2024-09-22T03:05:04.164793+0000","last_clean":"2024-09-22T03:05:04.164793+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:04.164793+0000","last_undegraded":"2024-09-22T03:05:04.164793+0000","last_fullsized":"2024-09-22T03:05:04.164793+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":282,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:04.181993+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:04.181993+0000","last_peered":"2024-09-22T03:05:04.181993+0000","last_clean":"2024-09-22T03:05:04.181993+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:04.181993+0000","last_undegraded":"2024-09-22T03:05:04.181993+0000","last_fullsized":"2024-09-22T03:05:04.181993+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"59'3925","reported_seq":4080,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:04.800093+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:05:04.800093+0000","last_peered":"2024-09-22T03:05:04.800093+0000","last_clean":"2024-09-22T03:05:04.800093+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:05:04.800093+0000","last_undegraded":"2024-09-22T03:05:04.800093+0000","last_fullsized":"2024-09-22T03:05:04.800093+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3926,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"59'3823","reported_seq":4917,"reported_epoch":60,"state":"peering","last_fresh":"2024-09-22T03:05:08.245991+0000","last_change":"2024-09-22T03:05:08.218079+0000","last_active":"2024-09-22T03:05:08.218079+0000","last_peered":"2024-09-22T03:05:08.141977+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:05:08.245991+0000","last_undegraded":"2024-09-22T03:05:08.245991+0000","last_fullsized":"2024-09-22T03:05:08.245991+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3823,"log_dups_size":0,"ondisk_log_size":3823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3823,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":116,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.190580+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:08.190580+0000","last_peered":"2024-09-22T03:05:08.190580+0000","last_clean":"2024-09-22T03:05:08.190580+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:08.190580+0000","last_undegraded":"2024-09-22T03:05:08.190580+0000","last_fullsized":"2024-09-22T03:05:08.190580+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"59'3821","reported_seq":3970,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:04.770733+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:05:04.770733+0000","last_peered":"2024-09-22T03:05:04.770733+0000","last_clean":"2024-09-22T03:05:04.770733+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:05:04.770733+0000","last_undegraded":"2024-09-22T03:05:04.770733+0000","last_fullsized":"2024-09-22T03:05:04.770733+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10092544,"num_objects":154,"num_object_clones":0,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3822,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":91,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.207259+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:08.207259+0000","last_peered":"2024-09-22T03:05:08.207259+0000","last_clean":"2024-09-22T03:05:08.207259+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:08.207259+0000","last_undegraded":"2024-09-22T03:05:08.207259+0000","last_fullsized":"2024-09-22T03:05:08.207259+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"60'3700","reported_seq":4752,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.539677+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:05:08.539677+0000","last_peered":"2024-09-22T03:05:08.539677+0000","last_clean":"2024-09-22T03:05:08.539677+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:05:08.539677+0000","last_undegraded":"2024-09-22T03:05:08.539677+0000","last_fullsized":"2024-09-22T03:05:08.539677+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3701,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":123,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.190667+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:08.190667+0000","last_peered":"2024-09-22T03:05:08.190667+0000","last_clean":"2024-09-22T03:05:08.190667+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:08.190667+0000","last_undegraded":"2024-09-22T03:05:08.190667+0000","last_fullsized":"2024-09-22T03:05:08.190667+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"60'3558","reported_seq":4645,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.709331+0000","last_change":"2024-09-22T03:05:00.209116+0000","last_active":"2024-09-22T03:05:08.709331+0000","last_peered":"2024-09-22T03:05:08.709331+0000","last_clean":"2024-09-22T03:05:08.709331+0000","last_became_active":"2024-09-22T03:05:00.203012+0000","last_became_peered":"2024-09-22T03:05:00.203012+0000","last_unstale":"2024-09-22T03:05:08.709331+0000","last_undegraded":"2024-09-22T03:05:08.709331+0000","last_fullsized":"2024-09-22T03:05:08.709331+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":125,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.189688+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:08.189688+0000","last_peered":"2024-09-22T03:05:08.189688+0000","last_clean":"2024-09-22T03:05:08.189688+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:08.189688+0000","last_undegraded":"2024-09-22T03:05:08.189688+0000","last_fullsized":"2024-09-22T03:05:08.189688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":110559256,"num_objects":1688,"num_object_clones":0,"num_object_copies":5064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":59724,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5094,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22028288,"data_stored":352256072,"data_compressed":16727000,"data_compressed_allocated":22016000,"data_compressed_original":352256000,"omap_allocated":0,"internal_metadata":0},"log_size":59710,"ondisk_log_size":59710,"up":48,"acting":47,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691930,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71676,"kb_used_data":8684,"kb_used_omap":0,"kb_used_meta":62976,"kb_avail":94300164,"statfs":{"total":96636764160,"available":96563367936,"internally_reserved":0,"allocated":8892416,"data_stored":134537661,"data_compressed":6379600,"data_compressed_allocated":8396800,"data_compressed_original":134348800,"omap_allocated":0,"internal_metadata":64487424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":63068,"kb_used_data":5452,"kb_used_omap":0,"kb_used_meta":57600,"kb_avail":94308772,"statfs":{"total":96636764160,"available":96572182528,"internally_reserved":0,"allocated":5582848,"data_stored":81129664,"data_compressed":3845271,"data_compressed_allocated":5062656,"data_compressed_original":80920576,"omap_allocated":0,"internal_metadata":58982400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673344,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76968,"kb_used_data":5720,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94294872,"statfs":{"total":96636764160,"available":96557948928,"internally_reserved":0,"allocated":5857280,"data_stored":85648330,"data_compressed":4059999,"data_compressed_allocated":5345280,"data_compressed_original":85442560,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888238,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62508,"kb_used_data":4956,"kb_used_omap":0,"kb_used_meta":57536,"kb_avail":94309332,"statfs":{"total":96636764160,"available":96572755968,"internally_reserved":0,"allocated":5074944,"data_stored":72937688,"data_compressed":3456271,"data_compressed_allocated":4550656,"data_compressed_original":72728576,"omap_allocated":0,"internal_metadata":58916864},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4911104,"data_stored":78512152,"data_compressed":3728176,"data_compressed_allocated":4907008,"data_compressed_original":78512128,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":68747288,"data_compressed":3264488,"data_compressed_allocated":4296704,"data_compressed_original":68747264,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4956160,"data_stored":79298560,"data_compressed":3765520,"data_compressed_allocated":4956160,"data_compressed_original":79298560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7860224,"data_stored":125698072,"data_compressed":5968816,"data_compressed_allocated":7856128,"data_compressed_original":125698048,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:09.274 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:05:09.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:09.402+0000 7efeff317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:09.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:09.408+0000 7efeff317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:09.890 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:09.890 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":395,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974243,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+undersized+remapped","count":2}],"num_pgs":25,"num_pools":3,"num_objects":2167,"data_bytes":142279243,"bytes_used":278933504,"bytes_avail":386268123136,"bytes_total":386547056640,"recovering_objects_per_sec":62,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":375,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":203},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:05:09.910 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:05:09.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:10.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:10.048+0000 7fca986b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:10.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:10.057+0000 7fca986b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:10.385 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:10.386 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:10.403 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-22T03:05:10.361774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95814219,"num_objects":1458,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":60097,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5094,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60044,"ondisk_log_size":60044,"up":75,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":258536,"kb_used_data":21608,"kb_used_omap":0,"kb_used_meta":236864,"kb_avail":377228824,"statfs":{"total":386547056640,"available":386282315776,"internally_reserved":0,"allocated":22126592,"data_stored":321503488,"data_compressed":15235981,"data_compressed_allocated":20058112,"data_compressed_original":320684032,"omap_allocated":0,"internal_metadata":242548736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-79298560,"num_objects":-1210,"num_object_clones":0,"num_object_copies":-3630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1210,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.207443"},"pg_stats":[{"pgid":"3.f","version":"59'3767","reported_seq":3939,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.096014+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:05:07.096014+0000","last_peered":"2024-09-22T03:05:07.096014+0000","last_clean":"2024-09-22T03:05:07.096014+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:05:07.096014+0000","last_undegraded":"2024-09-22T03:05:07.096014+0000","last_fullsized":"2024-09-22T03:05:07.096014+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7471104,"num_objects":114,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3768,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"60'3738","reported_seq":4374,"reported_epoch":60,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:05:08.820299+0000","last_change":"2024-09-22T03:04:59.208812+0000","last_active":"2024-09-22T03:05:08.820299+0000","last_peered":"2024-09-22T03:05:08.820299+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:05:08.820299+0000","last_undegraded":"2024-09-22T03:05:08.820299+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3738,"log_dups_size":0,"ondisk_log_size":3738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5767168,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3738,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":88}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"60'3767","reported_seq":3925,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.547949+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:05:08.547949+0000","last_peered":"2024-09-22T03:05:08.547949+0000","last_clean":"2024-09-22T03:05:08.547949+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:05:08.547949+0000","last_undegraded":"2024-09-22T03:05:08.547949+0000","last_fullsized":"2024-09-22T03:05:08.547949+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701656,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3768,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"59'3582","reported_seq":4677,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.085692+0000","last_change":"2024-09-22T03:05:04.213909+0000","last_active":"2024-09-22T03:05:07.085692+0000","last_peered":"2024-09-22T03:05:07.085692+0000","last_clean":"2024-09-22T03:05:07.085692+0000","last_became_active":"2024-09-22T03:05:04.207449+0000","last_became_peered":"2024-09-22T03:05:04.207449+0000","last_unstale":"2024-09-22T03:05:07.085692+0000","last_undegraded":"2024-09-22T03:05:07.085692+0000","last_fullsized":"2024-09-22T03:05:07.085692+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3583,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"61'3728","reported_seq":4822,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.888537+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:05:09.888537+0000","last_peered":"2024-09-22T03:05:09.888537+0000","last_clean":"2024-09-22T03:05:09.888537+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:05:09.888537+0000","last_undegraded":"2024-09-22T03:05:09.888537+0000","last_fullsized":"2024-09-22T03:05:09.888537+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3728,"log_dups_size":0,"ondisk_log_size":3728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5308416,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3729,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"59'3690","reported_seq":4829,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.262590+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:05:07.262590+0000","last_peered":"2024-09-22T03:05:07.262590+0000","last_clean":"2024-09-22T03:05:07.262590+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:05:07.262590+0000","last_undegraded":"2024-09-22T03:05:07.262590+0000","last_fullsized":"2024-09-22T03:05:07.262590+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6488064,"num_objects":99,"num_object_clones":0,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3691,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"59'3627","reported_seq":4700,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.017529+0000","last_change":"2024-09-22T03:05:02.795956+0000","last_active":"2024-09-22T03:05:07.017529+0000","last_peered":"2024-09-22T03:05:07.017529+0000","last_clean":"2024-09-22T03:05:07.017529+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:05:07.017529+0000","last_undegraded":"2024-09-22T03:05:07.017529+0000","last_fullsized":"2024-09-22T03:05:07.017529+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6684672,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3628,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"60'3808","reported_seq":4986,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.599559+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:05:08.599559+0000","last_peered":"2024-09-22T03:05:08.599559+0000","last_clean":"2024-09-22T03:05:08.599559+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:05:08.599559+0000","last_undegraded":"2024-09-22T03:05:08.599559+0000","last_fullsized":"2024-09-22T03:05:08.599559+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3808,"log_dups_size":0,"ondisk_log_size":3808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4784128,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":136,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.177832+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:09.177832+0000","last_peered":"2024-09-22T03:05:09.177832+0000","last_clean":"2024-09-22T03:05:09.177832+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:09.177832+0000","last_undegraded":"2024-09-22T03:05:09.177832+0000","last_fullsized":"2024-09-22T03:05:09.177832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"60'3778","reported_seq":3937,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.455212+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:05:08.455212+0000","last_peered":"2024-09-22T03:05:08.455212+0000","last_clean":"2024-09-22T03:05:08.455212+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:05:08.455212+0000","last_undegraded":"2024-09-22T03:05:08.455212+0000","last_fullsized":"2024-09-22T03:05:08.455212+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3779,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":159,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:04.179196+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:04.179196+0000","last_peered":"2024-09-22T03:05:04.179196+0000","last_clean":"2024-09-22T03:05:04.179196+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:04.179196+0000","last_undegraded":"2024-09-22T03:05:04.179196+0000","last_fullsized":"2024-09-22T03:05:04.179196+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"59'3809","reported_seq":5021,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.219637+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:05:08.219637+0000","last_peered":"2024-09-22T03:05:08.219637+0000","last_clean":"2024-09-22T03:05:08.219637+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:05:08.219637+0000","last_undegraded":"2024-09-22T03:05:08.219637+0000","last_fullsized":"2024-09-22T03:05:08.219637+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7077888,"num_objects":108,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3810,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":148,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.218821+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:08.218821+0000","last_peered":"2024-09-22T03:05:08.218821+0000","last_clean":"2024-09-22T03:05:08.218821+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:08.218821+0000","last_undegraded":"2024-09-22T03:05:08.218821+0000","last_fullsized":"2024-09-22T03:05:08.218821+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'3697","reported_seq":4768,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.984640+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:05:09.984640+0000","last_peered":"2024-09-22T03:05:09.984640+0000","last_clean":"2024-09-22T03:05:09.984640+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:05:09.984640+0000","last_undegraded":"2024-09-22T03:05:09.984640+0000","last_fullsized":"2024-09-22T03:05:09.984640+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4718592,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3698,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":146,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.176702+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:09.176702+0000","last_peered":"2024-09-22T03:05:09.176702+0000","last_clean":"2024-09-22T03:05:09.176702+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:09.176702+0000","last_undegraded":"2024-09-22T03:05:09.176702+0000","last_fullsized":"2024-09-22T03:05:09.176702+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":282,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:04.181993+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:04.181993+0000","last_peered":"2024-09-22T03:05:04.181993+0000","last_clean":"2024-09-22T03:05:04.181993+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:04.181993+0000","last_undegraded":"2024-09-22T03:05:04.181993+0000","last_fullsized":"2024-09-22T03:05:04.181993+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'3987","reported_seq":4148,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.992537+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:05:09.992537+0000","last_peered":"2024-09-22T03:05:09.992537+0000","last_clean":"2024-09-22T03:05:09.992537+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:05:09.992537+0000","last_undegraded":"2024-09-22T03:05:09.992537+0000","last_fullsized":"2024-09-22T03:05:09.992537+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3987,"log_dups_size":0,"ondisk_log_size":3987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"59'3823","reported_seq":4917,"reported_epoch":60,"state":"peering","last_fresh":"2024-09-22T03:05:08.245991+0000","last_change":"2024-09-22T03:05:08.218079+0000","last_active":"2024-09-22T03:05:08.218079+0000","last_peered":"2024-09-22T03:05:08.141977+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:05:08.245991+0000","last_undegraded":"2024-09-22T03:05:08.245991+0000","last_fullsized":"2024-09-22T03:05:08.245991+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3823,"log_dups_size":0,"ondisk_log_size":3823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3823,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":116,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.190580+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:08.190580+0000","last_peered":"2024-09-22T03:05:08.190580+0000","last_clean":"2024-09-22T03:05:08.190580+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:08.190580+0000","last_undegraded":"2024-09-22T03:05:08.190580+0000","last_fullsized":"2024-09-22T03:05:08.190580+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"61'3885","reported_seq":4040,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.913841+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:05:09.913841+0000","last_peered":"2024-09-22T03:05:09.913841+0000","last_clean":"2024-09-22T03:05:09.913841+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:05:09.913841+0000","last_undegraded":"2024-09-22T03:05:09.913841+0000","last_fullsized":"2024-09-22T03:05:09.913841+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3885,"log_dups_size":0,"ondisk_log_size":3885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5898240,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3886,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":91,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.207259+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:08.207259+0000","last_peered":"2024-09-22T03:05:08.207259+0000","last_clean":"2024-09-22T03:05:08.207259+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:08.207259+0000","last_undegraded":"2024-09-22T03:05:08.207259+0000","last_fullsized":"2024-09-22T03:05:08.207259+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"60'3700","reported_seq":4752,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.539677+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:05:08.539677+0000","last_peered":"2024-09-22T03:05:08.539677+0000","last_clean":"2024-09-22T03:05:08.539677+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:05:08.539677+0000","last_undegraded":"2024-09-22T03:05:08.539677+0000","last_fullsized":"2024-09-22T03:05:08.539677+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3701,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":123,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.190667+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:08.190667+0000","last_peered":"2024-09-22T03:05:08.190667+0000","last_clean":"2024-09-22T03:05:08.190667+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:08.190667+0000","last_undegraded":"2024-09-22T03:05:08.190667+0000","last_fullsized":"2024-09-22T03:05:08.190667+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"60'3558","reported_seq":4645,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.709331+0000","last_change":"2024-09-22T03:05:00.209116+0000","last_active":"2024-09-22T03:05:08.709331+0000","last_peered":"2024-09-22T03:05:08.709331+0000","last_clean":"2024-09-22T03:05:08.709331+0000","last_became_active":"2024-09-22T03:05:00.203012+0000","last_became_peered":"2024-09-22T03:05:00.203012+0000","last_unstale":"2024-09-22T03:05:08.709331+0000","last_undegraded":"2024-09-22T03:05:08.709331+0000","last_fullsized":"2024-09-22T03:05:08.709331+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":125,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.189688+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:08.189688+0000","last_peered":"2024-09-22T03:05:08.189688+0000","last_clean":"2024-09-22T03:05:08.189688+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:08.189688+0000","last_undegraded":"2024-09-22T03:05:08.189688+0000","last_fullsized":"2024-09-22T03:05:08.189688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":95223832,"num_objects":1454,"num_object_clones":0,"num_object_copies":4362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":59958,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5094,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18718720,"data_stored":299302984,"data_compressed":14212504,"data_compressed_allocated":18706432,"data_compressed_original":299302912,"omap_allocated":0,"internal_metadata":0},"log_size":59944,"ondisk_log_size":59944,"up":48,"acting":47,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55992,"kb_used_data":5480,"kb_used_omap":0,"kb_used_meta":50496,"kb_avail":94315848,"statfs":{"total":96636764160,"available":96579428352,"internally_reserved":0,"allocated":5611520,"data_stored":81787806,"data_compressed":3874440,"data_compressed_allocated":5099520,"data_compressed_original":81592320,"omap_allocated":0,"internal_metadata":51707904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":63068,"kb_used_data":5452,"kb_used_omap":0,"kb_used_meta":57600,"kb_avail":94308772,"statfs":{"total":96636764160,"available":96572182528,"internally_reserved":0,"allocated":5582848,"data_stored":81129664,"data_compressed":3845271,"data_compressed_allocated":5062656,"data_compressed_original":80920576,"omap_allocated":0,"internal_metadata":58982400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673344,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76968,"kb_used_data":5720,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94294872,"statfs":{"total":96636764160,"available":96557948928,"internally_reserved":0,"allocated":5857280,"data_stored":85648330,"data_compressed":4059999,"data_compressed_allocated":5345280,"data_compressed_original":85442560,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888238,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62508,"kb_used_data":4956,"kb_used_omap":0,"kb_used_meta":57536,"kb_avail":94309332,"statfs":{"total":96636764160,"available":96572755968,"internally_reserved":0,"allocated":5074944,"data_stored":72937688,"data_compressed":3456271,"data_compressed_allocated":4550656,"data_compressed_original":72728576,"omap_allocated":0,"internal_metadata":58916864},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4911104,"data_stored":78512152,"data_compressed":3728176,"data_compressed_allocated":4907008,"data_compressed_original":78512128,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":68747288,"data_compressed":3264488,"data_compressed_allocated":4296704,"data_compressed_original":68747264,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4956160,"data_stored":79298560,"data_compressed":3765520,"data_compressed_allocated":4956160,"data_compressed_original":79298560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4550656,"data_stored":72744984,"data_compressed":3454320,"data_compressed_allocated":4546560,"data_compressed_original":72744960,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:10.719 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:05:10.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:10.863+0000 7f2475797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:10.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:10.869+0000 7f2475797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:13.272 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:05:13.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:13.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:13.406+0000 7fc693917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:13.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:13.414+0000 7fc693917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:13.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:13.553+0000 7f602204f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:13.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:13.565+0000 7f602204f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:13.850 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:13.851 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:13.864 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-22T03:05:12.362202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95814219,"num_objects":1458,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":60097,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5094,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60044,"ondisk_log_size":60044,"up":75,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":258536,"kb_used_data":21608,"kb_used_omap":0,"kb_used_meta":236864,"kb_avail":377228824,"statfs":{"total":386547056640,"available":386282315776,"internally_reserved":0,"allocated":22126592,"data_stored":321503488,"data_compressed":15235981,"data_compressed_allocated":20058112,"data_compressed_original":320684032,"omap_allocated":0,"internal_metadata":242548736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-79298560,"num_objects":-1210,"num_object_clones":0,"num_object_copies":-3630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1210,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001578"},"pg_stats":[{"pgid":"3.f","version":"59'3767","reported_seq":3939,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.096014+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:05:07.096014+0000","last_peered":"2024-09-22T03:05:07.096014+0000","last_clean":"2024-09-22T03:05:07.096014+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:05:07.096014+0000","last_undegraded":"2024-09-22T03:05:07.096014+0000","last_fullsized":"2024-09-22T03:05:07.096014+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7471104,"num_objects":114,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3768,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"60'3738","reported_seq":4374,"reported_epoch":60,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:05:08.820299+0000","last_change":"2024-09-22T03:04:59.208812+0000","last_active":"2024-09-22T03:05:08.820299+0000","last_peered":"2024-09-22T03:05:08.820299+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:05:08.820299+0000","last_undegraded":"2024-09-22T03:05:08.820299+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3738,"log_dups_size":0,"ondisk_log_size":3738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5767168,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3738,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":88}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"60'3767","reported_seq":3925,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.547949+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:05:08.547949+0000","last_peered":"2024-09-22T03:05:08.547949+0000","last_clean":"2024-09-22T03:05:08.547949+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:05:08.547949+0000","last_undegraded":"2024-09-22T03:05:08.547949+0000","last_fullsized":"2024-09-22T03:05:08.547949+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701656,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3768,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"59'3582","reported_seq":4677,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.085692+0000","last_change":"2024-09-22T03:05:04.213909+0000","last_active":"2024-09-22T03:05:07.085692+0000","last_peered":"2024-09-22T03:05:07.085692+0000","last_clean":"2024-09-22T03:05:07.085692+0000","last_became_active":"2024-09-22T03:05:04.207449+0000","last_became_peered":"2024-09-22T03:05:04.207449+0000","last_unstale":"2024-09-22T03:05:07.085692+0000","last_undegraded":"2024-09-22T03:05:07.085692+0000","last_fullsized":"2024-09-22T03:05:07.085692+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3583,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"61'3728","reported_seq":4822,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.888537+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:05:09.888537+0000","last_peered":"2024-09-22T03:05:09.888537+0000","last_clean":"2024-09-22T03:05:09.888537+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:05:09.888537+0000","last_undegraded":"2024-09-22T03:05:09.888537+0000","last_fullsized":"2024-09-22T03:05:09.888537+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3728,"log_dups_size":0,"ondisk_log_size":3728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5308416,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3729,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"59'3690","reported_seq":4829,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.262590+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:05:07.262590+0000","last_peered":"2024-09-22T03:05:07.262590+0000","last_clean":"2024-09-22T03:05:07.262590+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:05:07.262590+0000","last_undegraded":"2024-09-22T03:05:07.262590+0000","last_fullsized":"2024-09-22T03:05:07.262590+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6488064,"num_objects":99,"num_object_clones":0,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3691,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"59'3627","reported_seq":4700,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:07.017529+0000","last_change":"2024-09-22T03:05:02.795956+0000","last_active":"2024-09-22T03:05:07.017529+0000","last_peered":"2024-09-22T03:05:07.017529+0000","last_clean":"2024-09-22T03:05:07.017529+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:05:07.017529+0000","last_undegraded":"2024-09-22T03:05:07.017529+0000","last_fullsized":"2024-09-22T03:05:07.017529+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6684672,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3628,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"60'3808","reported_seq":4986,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.599559+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:05:08.599559+0000","last_peered":"2024-09-22T03:05:08.599559+0000","last_clean":"2024-09-22T03:05:08.599559+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:05:08.599559+0000","last_undegraded":"2024-09-22T03:05:08.599559+0000","last_fullsized":"2024-09-22T03:05:08.599559+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3808,"log_dups_size":0,"ondisk_log_size":3808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4784128,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":136,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.177832+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:09.177832+0000","last_peered":"2024-09-22T03:05:09.177832+0000","last_clean":"2024-09-22T03:05:09.177832+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:09.177832+0000","last_undegraded":"2024-09-22T03:05:09.177832+0000","last_fullsized":"2024-09-22T03:05:09.177832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"60'3778","reported_seq":3937,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.455212+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:05:08.455212+0000","last_peered":"2024-09-22T03:05:08.455212+0000","last_clean":"2024-09-22T03:05:08.455212+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:05:08.455212+0000","last_undegraded":"2024-09-22T03:05:08.455212+0000","last_fullsized":"2024-09-22T03:05:08.455212+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3779,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":159,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:04.179196+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:04.179196+0000","last_peered":"2024-09-22T03:05:04.179196+0000","last_clean":"2024-09-22T03:05:04.179196+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:04.179196+0000","last_undegraded":"2024-09-22T03:05:04.179196+0000","last_fullsized":"2024-09-22T03:05:04.179196+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"59'3809","reported_seq":5021,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.219637+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:05:08.219637+0000","last_peered":"2024-09-22T03:05:08.219637+0000","last_clean":"2024-09-22T03:05:08.219637+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:05:08.219637+0000","last_undegraded":"2024-09-22T03:05:08.219637+0000","last_fullsized":"2024-09-22T03:05:08.219637+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7077888,"num_objects":108,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3810,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":148,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.218821+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:08.218821+0000","last_peered":"2024-09-22T03:05:08.218821+0000","last_clean":"2024-09-22T03:05:08.218821+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:08.218821+0000","last_undegraded":"2024-09-22T03:05:08.218821+0000","last_fullsized":"2024-09-22T03:05:08.218821+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'3697","reported_seq":4768,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.984640+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:05:09.984640+0000","last_peered":"2024-09-22T03:05:09.984640+0000","last_clean":"2024-09-22T03:05:09.984640+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:05:09.984640+0000","last_undegraded":"2024-09-22T03:05:09.984640+0000","last_fullsized":"2024-09-22T03:05:09.984640+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4718592,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3698,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":146,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.176702+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:09.176702+0000","last_peered":"2024-09-22T03:05:09.176702+0000","last_clean":"2024-09-22T03:05:09.176702+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:09.176702+0000","last_undegraded":"2024-09-22T03:05:09.176702+0000","last_fullsized":"2024-09-22T03:05:09.176702+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":282,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-22T03:05:04.181993+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:04.181993+0000","last_peered":"2024-09-22T03:05:04.181993+0000","last_clean":"2024-09-22T03:05:04.181993+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:04.181993+0000","last_undegraded":"2024-09-22T03:05:04.181993+0000","last_fullsized":"2024-09-22T03:05:04.181993+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'3987","reported_seq":4148,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.992537+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:05:09.992537+0000","last_peered":"2024-09-22T03:05:09.992537+0000","last_clean":"2024-09-22T03:05:09.992537+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:05:09.992537+0000","last_undegraded":"2024-09-22T03:05:09.992537+0000","last_fullsized":"2024-09-22T03:05:09.992537+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3987,"log_dups_size":0,"ondisk_log_size":3987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"59'3823","reported_seq":4917,"reported_epoch":60,"state":"peering","last_fresh":"2024-09-22T03:05:08.245991+0000","last_change":"2024-09-22T03:05:08.218079+0000","last_active":"2024-09-22T03:05:08.218079+0000","last_peered":"2024-09-22T03:05:08.141977+0000","last_clean":"2024-09-22T03:02:34.975870+0000","last_became_active":"2024-09-22T03:04:05.631683+0000","last_became_peered":"2024-09-22T03:04:05.631683+0000","last_unstale":"2024-09-22T03:05:08.245991+0000","last_undegraded":"2024-09-22T03:05:08.245991+0000","last_fullsized":"2024-09-22T03:05:08.245991+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3823,"log_dups_size":0,"ondisk_log_size":3823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3823,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":116,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.190580+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:08.190580+0000","last_peered":"2024-09-22T03:05:08.190580+0000","last_clean":"2024-09-22T03:05:08.190580+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:08.190580+0000","last_undegraded":"2024-09-22T03:05:08.190580+0000","last_fullsized":"2024-09-22T03:05:08.190580+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"61'3885","reported_seq":4040,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.913841+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:05:09.913841+0000","last_peered":"2024-09-22T03:05:09.913841+0000","last_clean":"2024-09-22T03:05:09.913841+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:05:09.913841+0000","last_undegraded":"2024-09-22T03:05:09.913841+0000","last_fullsized":"2024-09-22T03:05:09.913841+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3885,"log_dups_size":0,"ondisk_log_size":3885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5898240,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3886,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":91,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.207259+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:08.207259+0000","last_peered":"2024-09-22T03:05:08.207259+0000","last_clean":"2024-09-22T03:05:08.207259+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:08.207259+0000","last_undegraded":"2024-09-22T03:05:08.207259+0000","last_fullsized":"2024-09-22T03:05:08.207259+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"60'3700","reported_seq":4752,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.539677+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:05:08.539677+0000","last_peered":"2024-09-22T03:05:08.539677+0000","last_clean":"2024-09-22T03:05:08.539677+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:05:08.539677+0000","last_undegraded":"2024-09-22T03:05:08.539677+0000","last_fullsized":"2024-09-22T03:05:08.539677+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3701,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":123,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.190667+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:08.190667+0000","last_peered":"2024-09-22T03:05:08.190667+0000","last_clean":"2024-09-22T03:05:08.190667+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:08.190667+0000","last_undegraded":"2024-09-22T03:05:08.190667+0000","last_fullsized":"2024-09-22T03:05:08.190667+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"60'3558","reported_seq":4645,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.709331+0000","last_change":"2024-09-22T03:05:00.209116+0000","last_active":"2024-09-22T03:05:08.709331+0000","last_peered":"2024-09-22T03:05:08.709331+0000","last_clean":"2024-09-22T03:05:08.709331+0000","last_became_active":"2024-09-22T03:05:00.203012+0000","last_became_peered":"2024-09-22T03:05:00.203012+0000","last_unstale":"2024-09-22T03:05:08.709331+0000","last_undegraded":"2024-09-22T03:05:08.709331+0000","last_fullsized":"2024-09-22T03:05:08.709331+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":125,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-22T03:05:08.189688+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:08.189688+0000","last_peered":"2024-09-22T03:05:08.189688+0000","last_clean":"2024-09-22T03:05:08.189688+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:08.189688+0000","last_undegraded":"2024-09-22T03:05:08.189688+0000","last_fullsized":"2024-09-22T03:05:08.189688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":95223832,"num_objects":1454,"num_object_clones":0,"num_object_copies":4362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":59958,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5094,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18718720,"data_stored":299302984,"data_compressed":14212504,"data_compressed_allocated":18706432,"data_compressed_original":299302912,"omap_allocated":0,"internal_metadata":0},"log_size":59944,"ondisk_log_size":59944,"up":48,"acting":47,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55992,"kb_used_data":5480,"kb_used_omap":0,"kb_used_meta":50496,"kb_avail":94315848,"statfs":{"total":96636764160,"available":96579428352,"internally_reserved":0,"allocated":5611520,"data_stored":81787806,"data_compressed":3874440,"data_compressed_allocated":5099520,"data_compressed_original":81592320,"omap_allocated":0,"internal_metadata":51707904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":63068,"kb_used_data":5452,"kb_used_omap":0,"kb_used_meta":57600,"kb_avail":94308772,"statfs":{"total":96636764160,"available":96572182528,"internally_reserved":0,"allocated":5582848,"data_stored":81129664,"data_compressed":3845271,"data_compressed_allocated":5062656,"data_compressed_original":80920576,"omap_allocated":0,"internal_metadata":58982400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673344,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76968,"kb_used_data":5720,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94294872,"statfs":{"total":96636764160,"available":96557948928,"internally_reserved":0,"allocated":5857280,"data_stored":85648330,"data_compressed":4059999,"data_compressed_allocated":5345280,"data_compressed_original":85442560,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888238,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62508,"kb_used_data":4956,"kb_used_omap":0,"kb_used_meta":57536,"kb_avail":94309332,"statfs":{"total":96636764160,"available":96572755968,"internally_reserved":0,"allocated":5074944,"data_stored":72937688,"data_compressed":3456271,"data_compressed_allocated":4550656,"data_compressed_original":72728576,"omap_allocated":0,"internal_metadata":58916864},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4911104,"data_stored":78512152,"data_compressed":3728176,"data_compressed_allocated":4907008,"data_compressed_original":78512128,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":68747288,"data_compressed":3264488,"data_compressed_allocated":4296704,"data_compressed_original":68747264,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4956160,"data_stored":79298560,"data_compressed":3765520,"data_compressed_allocated":4956160,"data_compressed_original":79298560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4550656,"data_stored":72744984,"data_compressed":3454320,"data_compressed_allocated":4546560,"data_compressed_original":72744960,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:13.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:05:14.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:14.044+0000 7f3a15797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:14.051+0000 7f3a15797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:14.542 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:14.542 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":400,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974243,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":24},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":737,"data_bytes":48562763,"bytes_used":264511488,"bytes_avail":386282545152,"bytes_total":386547056640,"recovering_objects_per_sec":41,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":330,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":207},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:05:14.557 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:05:14.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:14.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:14.692+0000 7f51f8357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:14.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:14.700+0000 7f51f8357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:14.954 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:14.955 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:14.971 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-22T03:05:14.362572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":48562763,"num_objects":737,"num_object_clones":0,"num_object_copies":2211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":60818,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5238,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60763,"ondisk_log_size":60763,"up":75,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":258312,"kb_used_data":13064,"kb_used_omap":0,"kb_used_meta":245184,"kb_avail":377229048,"statfs":{"total":386547056640,"available":386282545152,"internally_reserved":0,"allocated":13377536,"data_stored":181007533,"data_compressed":8563853,"data_compressed_allocated":11276288,"data_compressed_original":180174848,"omap_allocated":0,"internal_metadata":251068416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-108724224,"num_objects":-1659,"num_object_clones":0,"num_object_copies":-4977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1659,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":330,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001613"},"pg_stats":[{"pgid":"3.f","version":"61'3832","reported_seq":4010,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:12.168743+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:05:12.168743+0000","last_peered":"2024-09-22T03:05:12.168743+0000","last_clean":"2024-09-22T03:05:12.168743+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:05:12.168743+0000","last_undegraded":"2024-09-22T03:05:12.168743+0000","last_fullsized":"2024-09-22T03:05:12.168743+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3832,"log_dups_size":0,"ondisk_log_size":3832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"61'3800","reported_seq":4727,"reported_epoch":61,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:05:13.821955+0000","last_change":"2024-09-22T03:04:59.208812+0000","last_active":"2024-09-22T03:05:13.821955+0000","last_peered":"2024-09-22T03:05:13.821955+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:05:13.821955+0000","last_undegraded":"2024-09-22T03:05:13.821955+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3800,"log_dups_size":0,"ondisk_log_size":3800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3801,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":394,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":25}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"61'3827","reported_seq":3988,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.743374+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:05:13.743374+0000","last_peered":"2024-09-22T03:05:13.743374+0000","last_clean":"2024-09-22T03:05:13.743374+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:05:13.743374+0000","last_undegraded":"2024-09-22T03:05:13.743374+0000","last_fullsized":"2024-09-22T03:05:13.743374+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769496,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3828,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"61'3642","reported_seq":4743,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:12.225433+0000","last_change":"2024-09-22T03:05:04.213909+0000","last_active":"2024-09-22T03:05:12.225433+0000","last_peered":"2024-09-22T03:05:12.225433+0000","last_clean":"2024-09-22T03:05:12.225433+0000","last_became_active":"2024-09-22T03:05:04.207449+0000","last_became_peered":"2024-09-22T03:05:04.207449+0000","last_unstale":"2024-09-22T03:05:12.225433+0000","last_undegraded":"2024-09-22T03:05:12.225433+0000","last_fullsized":"2024-09-22T03:05:12.225433+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3642,"log_dups_size":0,"ondisk_log_size":3642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3342336,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3643,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"61'3728","reported_seq":4822,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.888537+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:05:09.888537+0000","last_peered":"2024-09-22T03:05:09.888537+0000","last_clean":"2024-09-22T03:05:09.888537+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:05:09.888537+0000","last_undegraded":"2024-09-22T03:05:09.888537+0000","last_fullsized":"2024-09-22T03:05:09.888537+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3728,"log_dups_size":0,"ondisk_log_size":3728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5308416,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3729,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"61'3740","reported_seq":4885,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:12.148213+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:05:12.148213+0000","last_peered":"2024-09-22T03:05:12.148213+0000","last_clean":"2024-09-22T03:05:12.148213+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:05:12.148213+0000","last_undegraded":"2024-09-22T03:05:12.148213+0000","last_fullsized":"2024-09-22T03:05:12.148213+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3740,"log_dups_size":0,"ondisk_log_size":3740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3741,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"61'3683","reported_seq":4762,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:12.272773+0000","last_change":"2024-09-22T03:05:02.795956+0000","last_active":"2024-09-22T03:05:12.272773+0000","last_peered":"2024-09-22T03:05:12.272773+0000","last_clean":"2024-09-22T03:05:12.272773+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:05:12.272773+0000","last_undegraded":"2024-09-22T03:05:12.272773+0000","last_fullsized":"2024-09-22T03:05:12.272773+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3684,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"61'3854","reported_seq":5035,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.757099+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:05:13.757099+0000","last_peered":"2024-09-22T03:05:13.757099+0000","last_clean":"2024-09-22T03:05:13.757099+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:05:13.757099+0000","last_undegraded":"2024-09-22T03:05:13.757099+0000","last_fullsized":"2024-09-22T03:05:13.757099+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3854,"log_dups_size":0,"ondisk_log_size":3854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3855,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":136,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.177832+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:09.177832+0000","last_peered":"2024-09-22T03:05:09.177832+0000","last_clean":"2024-09-22T03:05:09.177832+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:09.177832+0000","last_undegraded":"2024-09-22T03:05:09.177832+0000","last_fullsized":"2024-09-22T03:05:09.177832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"61'3839","reported_seq":4001,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.556787+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:05:13.556787+0000","last_peered":"2024-09-22T03:05:13.556787+0000","last_clean":"2024-09-22T03:05:13.556787+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:05:13.556787+0000","last_undegraded":"2024-09-22T03:05:13.556787+0000","last_fullsized":"2024-09-22T03:05:13.556787+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3839,"log_dups_size":0,"ondisk_log_size":3839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3840,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":165,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.176362+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:09.176362+0000","last_peered":"2024-09-22T03:05:09.176362+0000","last_clean":"2024-09-22T03:05:09.176362+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:09.176362+0000","last_undegraded":"2024-09-22T03:05:09.176362+0000","last_fullsized":"2024-09-22T03:05:09.176362+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'3879","reported_seq":5094,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.733857+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:05:13.733857+0000","last_peered":"2024-09-22T03:05:13.733857+0000","last_clean":"2024-09-22T03:05:13.733857+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:05:13.733857+0000","last_undegraded":"2024-09-22T03:05:13.733857+0000","last_fullsized":"2024-09-22T03:05:13.733857+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3879,"log_dups_size":0,"ondisk_log_size":3879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3880,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":151,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.183652+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:09.183652+0000","last_peered":"2024-09-22T03:05:09.183652+0000","last_clean":"2024-09-22T03:05:09.183652+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:09.183652+0000","last_undegraded":"2024-09-22T03:05:09.183652+0000","last_fullsized":"2024-09-22T03:05:09.183652+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'3697","reported_seq":4768,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.984640+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:05:09.984640+0000","last_peered":"2024-09-22T03:05:09.984640+0000","last_clean":"2024-09-22T03:05:09.984640+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:05:09.984640+0000","last_undegraded":"2024-09-22T03:05:09.984640+0000","last_fullsized":"2024-09-22T03:05:09.984640+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4718592,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3698,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":146,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.176702+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:09.176702+0000","last_peered":"2024-09-22T03:05:09.176702+0000","last_clean":"2024-09-22T03:05:09.176702+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:09.176702+0000","last_undegraded":"2024-09-22T03:05:09.176702+0000","last_fullsized":"2024-09-22T03:05:09.176702+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":288,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.179432+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:09.179432+0000","last_peered":"2024-09-22T03:05:09.179432+0000","last_clean":"2024-09-22T03:05:09.179432+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:09.179432+0000","last_undegraded":"2024-09-22T03:05:09.179432+0000","last_fullsized":"2024-09-22T03:05:09.179432+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'3987","reported_seq":4148,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.992537+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:05:09.992537+0000","last_peered":"2024-09-22T03:05:09.992537+0000","last_clean":"2024-09-22T03:05:09.992537+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:05:09.992537+0000","last_undegraded":"2024-09-22T03:05:09.992537+0000","last_fullsized":"2024-09-22T03:05:09.992537+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3987,"log_dups_size":0,"ondisk_log_size":3987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"61'3894","reported_seq":4997,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.445755+0000","last_change":"2024-09-22T03:05:09.199730+0000","last_active":"2024-09-22T03:05:13.445755+0000","last_peered":"2024-09-22T03:05:13.445755+0000","last_clean":"2024-09-22T03:05:13.445755+0000","last_became_active":"2024-09-22T03:05:09.195261+0000","last_became_peered":"2024-09-22T03:05:09.195261+0000","last_unstale":"2024-09-22T03:05:13.445755+0000","last_undegraded":"2024-09-22T03:05:13.445755+0000","last_fullsized":"2024-09-22T03:05:13.445755+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3895,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":119,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.177843+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:09.177843+0000","last_peered":"2024-09-22T03:05:09.177843+0000","last_clean":"2024-09-22T03:05:09.177843+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:09.177843+0000","last_undegraded":"2024-09-22T03:05:09.177843+0000","last_fullsized":"2024-09-22T03:05:09.177843+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"61'3885","reported_seq":4040,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.913841+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:05:09.913841+0000","last_peered":"2024-09-22T03:05:09.913841+0000","last_clean":"2024-09-22T03:05:09.913841+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:05:09.913841+0000","last_undegraded":"2024-09-22T03:05:09.913841+0000","last_fullsized":"2024-09-22T03:05:09.913841+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3885,"log_dups_size":0,"ondisk_log_size":3885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5898240,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3886,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.180549+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:09.180549+0000","last_peered":"2024-09-22T03:05:09.180549+0000","last_clean":"2024-09-22T03:05:09.180549+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:09.180549+0000","last_undegraded":"2024-09-22T03:05:09.180549+0000","last_fullsized":"2024-09-22T03:05:09.180549+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"61'3764","reported_seq":4819,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.784013+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:05:13.784013+0000","last_peered":"2024-09-22T03:05:13.784013+0000","last_clean":"2024-09-22T03:05:13.784013+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:05:13.784013+0000","last_undegraded":"2024-09-22T03:05:13.784013+0000","last_fullsized":"2024-09-22T03:05:13.784013+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":126,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.178848+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:09.178848+0000","last_peered":"2024-09-22T03:05:09.178848+0000","last_clean":"2024-09-22T03:05:09.178848+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:09.178848+0000","last_undegraded":"2024-09-22T03:05:09.178848+0000","last_fullsized":"2024-09-22T03:05:09.178848+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'3612","reported_seq":4702,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.238913+0000","last_change":"2024-09-22T03:05:00.209116+0000","last_active":"2024-09-22T03:05:13.238913+0000","last_peered":"2024-09-22T03:05:13.238913+0000","last_clean":"2024-09-22T03:05:13.238913+0000","last_became_active":"2024-09-22T03:05:00.203012+0000","last_became_peered":"2024-09-22T03:05:00.203012+0000","last_unstale":"2024-09-22T03:05:13.238913+0000","last_undegraded":"2024-09-22T03:05:13.238913+0000","last_fullsized":"2024-09-22T03:05:13.238913+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":128,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.179852+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:09.179852+0000","last_peered":"2024-09-22T03:05:09.179852+0000","last_clean":"2024-09-22T03:05:09.179852+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:09.179852+0000","last_undegraded":"2024-09-22T03:05:09.179852+0000","last_fullsized":"2024-09-22T03:05:09.179852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":47972376,"num_objects":733,"num_object_clones":0,"num_object_copies":2199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":60679,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5238,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9814016,"data_stored":156827720,"data_compressed":7447016,"data_compressed_allocated":9801728,"data_compressed_original":156827648,"omap_allocated":0,"internal_metadata":0},"log_size":60663,"ondisk_log_size":60663,"up":48,"acting":47,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55992,"kb_used_data":5480,"kb_used_omap":0,"kb_used_meta":50496,"kb_avail":94315848,"statfs":{"total":96636764160,"available":96579428352,"internally_reserved":0,"allocated":5611520,"data_stored":81787806,"data_compressed":3874440,"data_compressed_allocated":5099520,"data_compressed_original":81592320,"omap_allocated":0,"internal_metadata":51707904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626502,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64104,"kb_used_data":2328,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94307736,"statfs":{"total":96636764160,"available":96571121664,"internally_reserved":0,"allocated":2383872,"data_stored":29818278,"data_compressed":1408575,"data_compressed_allocated":1855488,"data_compressed_original":29605888,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673349,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78500,"kb_used_data":3092,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94293340,"statfs":{"total":96636764160,"available":96556380160,"internally_reserved":0,"allocated":3166208,"data_stored":42335659,"data_compressed":2002967,"data_compressed_allocated":2637824,"data_compressed_original":42123264,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888243,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":59716,"kb_used_data":2164,"kb_used_omap":0,"kb_used_meta":57536,"kb_avail":94312124,"statfs":{"total":96636764160,"available":96575614976,"internally_reserved":0,"allocated":2215936,"data_stored":27065790,"data_compressed":1277871,"data_compressed_allocated":1683456,"data_compressed_original":26853376,"omap_allocated":0,"internal_metadata":58916864},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2191360,"data_stored":34996248,"data_compressed":1661808,"data_compressed_allocated":2187264,"data_compressed_original":34996224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1355776,"data_stored":21626904,"data_compressed":1026960,"data_compressed_allocated":1351680,"data_compressed_original":21626880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1716224,"data_stored":27459584,"data_compressed":1303928,"data_compressed_allocated":1716224,"data_compressed_original":27459584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4550656,"data_stored":72744984,"data_compressed":3454320,"data_compressed_allocated":4546560,"data_compressed_original":72744960,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:15.814 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:05:15.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:15.955+0000 7fedf0d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:15.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:15.961+0000 7fedf0d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:16.154 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Removed 30705 objects 2024-09-22T03:05:16.450 INFO:tasks.radosbench.radosbench.0.smithi081.stdout:Clean up completed and total clean up time :216.366 2024-09-22T03:05:17.498 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-22T03:05:17.499 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-22T03:05:17.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:17.604+0000 7f2b7846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:17.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:17.615+0000 7f2b7846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:17.974 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:18.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:18.074+0000 7f514e2a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:18.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:18.081+0000 7f514e2a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:18.261 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:18.261 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:18.273 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-22T03:05:17.427897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19333707,"num_objects":291,"num_object_clones":0,"num_object_copies":873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":61264,"num_write_kb":1967017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5238,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61209,"ondisk_log_size":61209,"up":75,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":256232,"kb_used_data":6824,"kb_used_omap":0,"kb_used_meta":249344,"kb_avail":377231128,"statfs":{"total":386547056640,"available":386284675072,"internally_reserved":0,"allocated":6987776,"data_stored":78836885,"data_compressed":3712245,"data_compressed_allocated":4890624,"data_compressed_original":78004224,"omap_allocated":0,"internal_metadata":255328256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-122945536,"num_objects":-1876,"num_object_clones":0,"num_object_copies":-5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1876,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.066512"},"pg_stats":[{"pgid":"3.f","version":"61'3881","reported_seq":4059,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:15.899427+0000","last_change":"2024-09-22T03:03:35.474091+0000","last_active":"2024-09-22T03:05:15.899427+0000","last_peered":"2024-09-22T03:05:15.899427+0000","last_clean":"2024-09-22T03:05:15.899427+0000","last_became_active":"2024-09-22T03:03:35.238621+0000","last_became_peered":"2024-09-22T03:03:35.238621+0000","last_unstale":"2024-09-22T03:05:15.899427+0000","last_undegraded":"2024-09-22T03:05:15.899427+0000","last_fullsized":"2024-09-22T03:05:15.899427+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3881,"log_dups_size":0,"ondisk_log_size":3881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3882,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"61'3800","reported_seq":4727,"reported_epoch":61,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-22T03:05:13.821955+0000","last_change":"2024-09-22T03:04:59.208812+0000","last_active":"2024-09-22T03:05:13.821955+0000","last_peered":"2024-09-22T03:05:13.821955+0000","last_clean":"2024-09-22T03:02:34.801026+0000","last_became_active":"2024-09-22T03:04:05.573595+0000","last_became_peered":"2024-09-22T03:04:05.573595+0000","last_unstale":"2024-09-22T03:05:13.821955+0000","last_undegraded":"2024-09-22T03:05:13.821955+0000","last_fullsized":"2024-09-22T03:04:05.330111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3800,"log_dups_size":0,"ondisk_log_size":3800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3801,"num_write_kb":122432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":394,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":25}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"61'3827","reported_seq":3988,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.743374+0000","last_change":"2024-09-22T03:03:35.496506+0000","last_active":"2024-09-22T03:05:13.743374+0000","last_peered":"2024-09-22T03:05:13.743374+0000","last_clean":"2024-09-22T03:05:13.743374+0000","last_became_active":"2024-09-22T03:03:35.238649+0000","last_became_peered":"2024-09-22T03:03:35.238649+0000","last_unstale":"2024-09-22T03:05:13.743374+0000","last_undegraded":"2024-09-22T03:05:13.743374+0000","last_fullsized":"2024-09-22T03:05:13.743374+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769496,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3828,"num_write_kb":123329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"61'3693","reported_seq":4794,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:15.989331+0000","last_change":"2024-09-22T03:05:04.213909+0000","last_active":"2024-09-22T03:05:15.989331+0000","last_peered":"2024-09-22T03:05:15.989331+0000","last_clean":"2024-09-22T03:05:15.989331+0000","last_became_active":"2024-09-22T03:05:04.207449+0000","last_became_peered":"2024-09-22T03:05:04.207449+0000","last_unstale":"2024-09-22T03:05:15.989331+0000","last_undegraded":"2024-09-22T03:05:15.989331+0000","last_fullsized":"2024-09-22T03:05:15.989331+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3693,"log_dups_size":0,"ondisk_log_size":3693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3694,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"61'3786","reported_seq":4880,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:15.002887+0000","last_change":"2024-09-22T03:04:45.149599+0000","last_active":"2024-09-22T03:05:15.002887+0000","last_peered":"2024-09-22T03:05:15.002887+0000","last_clean":"2024-09-22T03:05:15.002887+0000","last_became_active":"2024-09-22T03:04:45.140760+0000","last_became_peered":"2024-09-22T03:04:45.140760+0000","last_unstale":"2024-09-22T03:05:15.002887+0000","last_undegraded":"2024-09-22T03:05:15.002887+0000","last_fullsized":"2024-09-22T03:05:15.002887+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3787,"num_write_kb":121920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":475,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"61'3789","reported_seq":4934,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:16.003102+0000","last_change":"2024-09-22T03:04:29.220985+0000","last_active":"2024-09-22T03:05:16.003102+0000","last_peered":"2024-09-22T03:05:16.003102+0000","last_clean":"2024-09-22T03:05:16.003102+0000","last_became_active":"2024-09-22T03:04:29.208534+0000","last_became_peered":"2024-09-22T03:04:29.208534+0000","last_unstale":"2024-09-22T03:05:16.003102+0000","last_undegraded":"2024-09-22T03:05:16.003102+0000","last_fullsized":"2024-09-22T03:05:16.003102+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3790,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"61'3729","reported_seq":4808,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:15.828418+0000","last_change":"2024-09-22T03:05:02.795956+0000","last_active":"2024-09-22T03:05:15.828418+0000","last_peered":"2024-09-22T03:05:15.828418+0000","last_clean":"2024-09-22T03:05:15.828418+0000","last_became_active":"2024-09-22T03:04:27.249677+0000","last_became_peered":"2024-09-22T03:04:27.249677+0000","last_unstale":"2024-09-22T03:05:15.828418+0000","last_undegraded":"2024-09-22T03:05:15.828418+0000","last_fullsized":"2024-09-22T03:05:15.828418+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3729,"log_dups_size":0,"ondisk_log_size":3729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3730,"num_write_kb":119360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"61'3854","reported_seq":5035,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.757099+0000","last_change":"2024-09-22T03:04:28.200163+0000","last_active":"2024-09-22T03:05:13.757099+0000","last_peered":"2024-09-22T03:05:13.757099+0000","last_clean":"2024-09-22T03:05:13.757099+0000","last_became_active":"2024-09-22T03:04:28.192701+0000","last_became_peered":"2024-09-22T03:04:28.192701+0000","last_unstale":"2024-09-22T03:05:13.757099+0000","last_undegraded":"2024-09-22T03:05:13.757099+0000","last_fullsized":"2024-09-22T03:05:13.757099+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3854,"log_dups_size":0,"ondisk_log_size":3854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3855,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":136,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.177832+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:09.177832+0000","last_peered":"2024-09-22T03:05:09.177832+0000","last_clean":"2024-09-22T03:05:09.177832+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:09.177832+0000","last_undegraded":"2024-09-22T03:05:09.177832+0000","last_fullsized":"2024-09-22T03:05:09.177832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"61'3839","reported_seq":4001,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.556787+0000","last_change":"2024-09-22T03:03:35.503228+0000","last_active":"2024-09-22T03:05:13.556787+0000","last_peered":"2024-09-22T03:05:13.556787+0000","last_clean":"2024-09-22T03:05:13.556787+0000","last_became_active":"2024-09-22T03:03:35.234430+0000","last_became_peered":"2024-09-22T03:03:35.234430+0000","last_unstale":"2024-09-22T03:05:13.556787+0000","last_undegraded":"2024-09-22T03:05:13.556787+0000","last_fullsized":"2024-09-22T03:05:13.556787+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3839,"log_dups_size":0,"ondisk_log_size":3839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3840,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":165,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.176362+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:09.176362+0000","last_peered":"2024-09-22T03:05:09.176362+0000","last_clean":"2024-09-22T03:05:09.176362+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:09.176362+0000","last_undegraded":"2024-09-22T03:05:09.176362+0000","last_fullsized":"2024-09-22T03:05:09.176362+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"61'3879","reported_seq":5094,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.733857+0000","last_change":"2024-09-22T03:04:29.216122+0000","last_active":"2024-09-22T03:05:13.733857+0000","last_peered":"2024-09-22T03:05:13.733857+0000","last_clean":"2024-09-22T03:05:13.733857+0000","last_became_active":"2024-09-22T03:04:29.204641+0000","last_became_peered":"2024-09-22T03:04:29.204641+0000","last_unstale":"2024-09-22T03:05:13.733857+0000","last_undegraded":"2024-09-22T03:05:13.733857+0000","last_fullsized":"2024-09-22T03:05:13.733857+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3879,"log_dups_size":0,"ondisk_log_size":3879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3880,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":151,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.183652+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:09.183652+0000","last_peered":"2024-09-22T03:05:09.183652+0000","last_clean":"2024-09-22T03:05:09.183652+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:09.183652+0000","last_undegraded":"2024-09-22T03:05:09.183652+0000","last_fullsized":"2024-09-22T03:05:09.183652+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'3758","reported_seq":4829,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:14.906920+0000","last_change":"2024-09-22T03:04:46.164628+0000","last_active":"2024-09-22T03:05:14.906920+0000","last_peered":"2024-09-22T03:05:14.906920+0000","last_clean":"2024-09-22T03:05:14.906920+0000","last_became_active":"2024-09-22T03:04:46.154172+0000","last_became_peered":"2024-09-22T03:04:46.154172+0000","last_unstale":"2024-09-22T03:05:14.906920+0000","last_undegraded":"2024-09-22T03:05:14.906920+0000","last_fullsized":"2024-09-22T03:05:14.906920+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3758,"log_dups_size":0,"ondisk_log_size":3758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3759,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":146,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.176702+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:09.176702+0000","last_peered":"2024-09-22T03:05:09.176702+0000","last_clean":"2024-09-22T03:05:09.176702+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:09.176702+0000","last_undegraded":"2024-09-22T03:05:09.176702+0000","last_fullsized":"2024-09-22T03:05:09.176702+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":288,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.179432+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:09.179432+0000","last_peered":"2024-09-22T03:05:09.179432+0000","last_clean":"2024-09-22T03:05:09.179432+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:09.179432+0000","last_undegraded":"2024-09-22T03:05:09.179432+0000","last_fullsized":"2024-09-22T03:05:09.179432+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'4042","reported_seq":4203,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:14.744375+0000","last_change":"2024-09-22T03:03:35.544122+0000","last_active":"2024-09-22T03:05:14.744375+0000","last_peered":"2024-09-22T03:05:14.744375+0000","last_clean":"2024-09-22T03:05:14.744375+0000","last_became_active":"2024-09-22T03:03:35.267454+0000","last_became_peered":"2024-09-22T03:03:35.267454+0000","last_unstale":"2024-09-22T03:05:14.744375+0000","last_undegraded":"2024-09-22T03:05:14.744375+0000","last_fullsized":"2024-09-22T03:05:14.744375+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":4042,"log_dups_size":0,"ondisk_log_size":4042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":129856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"61'3894","reported_seq":4997,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.445755+0000","last_change":"2024-09-22T03:05:09.199730+0000","last_active":"2024-09-22T03:05:13.445755+0000","last_peered":"2024-09-22T03:05:13.445755+0000","last_clean":"2024-09-22T03:05:13.445755+0000","last_became_active":"2024-09-22T03:05:09.195261+0000","last_became_peered":"2024-09-22T03:05:09.195261+0000","last_unstale":"2024-09-22T03:05:13.445755+0000","last_undegraded":"2024-09-22T03:05:13.445755+0000","last_fullsized":"2024-09-22T03:05:13.445755+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3895,"num_write_kb":125568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":119,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.177843+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:09.177843+0000","last_peered":"2024-09-22T03:05:09.177843+0000","last_clean":"2024-09-22T03:05:09.177843+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:09.177843+0000","last_undegraded":"2024-09-22T03:05:09.177843+0000","last_fullsized":"2024-09-22T03:05:09.177843+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"61'3962","reported_seq":4117,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:14.775135+0000","last_change":"2024-09-22T03:03:35.497008+0000","last_active":"2024-09-22T03:05:14.775135+0000","last_peered":"2024-09-22T03:05:14.775135+0000","last_clean":"2024-09-22T03:05:14.775135+0000","last_became_active":"2024-09-22T03:03:35.283228+0000","last_became_peered":"2024-09-22T03:03:35.283228+0000","last_unstale":"2024-09-22T03:05:14.775135+0000","last_undegraded":"2024-09-22T03:05:14.775135+0000","last_fullsized":"2024-09-22T03:05:14.775135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3963,"num_write_kb":127232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.180549+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:09.180549+0000","last_peered":"2024-09-22T03:05:09.180549+0000","last_clean":"2024-09-22T03:05:09.180549+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:09.180549+0000","last_undegraded":"2024-09-22T03:05:09.180549+0000","last_fullsized":"2024-09-22T03:05:09.180549+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"61'3764","reported_seq":4819,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.784013+0000","last_change":"2024-09-22T03:04:44.343037+0000","last_active":"2024-09-22T03:05:13.784013+0000","last_peered":"2024-09-22T03:05:13.784013+0000","last_clean":"2024-09-22T03:05:13.784013+0000","last_became_active":"2024-09-22T03:04:44.210407+0000","last_became_peered":"2024-09-22T03:04:44.210407+0000","last_unstale":"2024-09-22T03:05:13.784013+0000","last_undegraded":"2024-09-22T03:05:13.784013+0000","last_fullsized":"2024-09-22T03:05:13.784013+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3765,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":126,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.178848+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:09.178848+0000","last_peered":"2024-09-22T03:05:09.178848+0000","last_clean":"2024-09-22T03:05:09.178848+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:09.178848+0000","last_undegraded":"2024-09-22T03:05:09.178848+0000","last_fullsized":"2024-09-22T03:05:09.178848+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'3612","reported_seq":4702,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:13.238913+0000","last_change":"2024-09-22T03:05:00.209116+0000","last_active":"2024-09-22T03:05:13.238913+0000","last_peered":"2024-09-22T03:05:13.238913+0000","last_clean":"2024-09-22T03:05:13.238913+0000","last_became_active":"2024-09-22T03:05:00.203012+0000","last_became_peered":"2024-09-22T03:05:00.203012+0000","last_unstale":"2024-09-22T03:05:13.238913+0000","last_undegraded":"2024-09-22T03:05:13.238913+0000","last_fullsized":"2024-09-22T03:05:13.238913+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:54.663941+0000","last_clean_scrub_stamp":"2024-09-22T02:58:54.663941+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":116608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":128,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.179852+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:09.179852+0000","last_peered":"2024-09-22T03:05:09.179852+0000","last_clean":"2024-09-22T03:05:09.179852+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:09.179852+0000","last_undegraded":"2024-09-22T03:05:09.179852+0000","last_fullsized":"2024-09-22T03:05:09.179852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":18743320,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":61125,"num_write_kb":1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5238,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3891200,"data_stored":62128176,"data_compressed":2950176,"data_compressed_allocated":3883008,"data_compressed_original":62128128,"omap_allocated":0,"internal_metadata":0},"log_size":61109,"ondisk_log_size":61109,"up":48,"acting":47,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691940,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56452,"kb_used_data":1780,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94315388,"statfs":{"total":96636764160,"available":96578957312,"internally_reserved":0,"allocated":1822720,"data_stored":21167006,"data_compressed":995840,"data_compressed_allocated":1310720,"data_compressed_original":20971520,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626502,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64104,"kb_used_data":2328,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94307736,"statfs":{"total":96636764160,"available":96571121664,"internally_reserved":0,"allocated":2383872,"data_stored":29818278,"data_compressed":1408575,"data_compressed_allocated":1855488,"data_compressed_original":29605888,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673354,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75960,"kb_used_data":552,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94295880,"statfs":{"total":96636764160,"available":96558981120,"internally_reserved":0,"allocated":565248,"data_stored":785811,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888243,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":59716,"kb_used_data":2164,"kb_used_omap":0,"kb_used_meta":57536,"kb_avail":94312124,"statfs":{"total":96636764160,"available":96575614976,"internally_reserved":0,"allocated":2215936,"data_stored":27065790,"data_compressed":1277871,"data_compressed_allocated":1683456,"data_compressed_original":26853376,"omap_allocated":0,"internal_metadata":58916864},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1355776,"data_stored":21626904,"data_compressed":1026960,"data_compressed_allocated":1351680,"data_compressed_original":21626880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1716224,"data_stored":27459584,"data_compressed":1303928,"data_compressed_allocated":1716224,"data_compressed_original":27459584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":819200,"data_stored":13041688,"data_compressed":619288,"data_compressed_allocated":815104,"data_compressed_original":13041664,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:18.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:05:18.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:18.373+0000 7fb366218640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:18.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:18.378+0000 7fb366218640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:18.436 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'unique_pool_0' removed 2024-09-22T03:05:18.453 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:05:18.454 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-22T03:05:18.462 INFO:tasks.thrashosds:joining thrashosds 2024-09-22T03:05:18.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:18.615+0000 7f635166c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:18.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:18.621+0000 7f635166c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:18.668 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:18.668 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":404,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":63,"num_osds":4,"num_up_osds":4,"osd_up_since":1726974243,"num_in_osds":4,"osd_in_since":1726973916,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":24},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":291,"data_bytes":19333707,"bytes_used":262381568,"bytes_avail":386284675072,"bytes_total":386547056640,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":152},"fsmap":{"epoch":1,"btime":"2024-09-22T02:58:34:075721+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T02:59:40.295712+0000","services":{}},"progress_events":{}} 2024-09-22T03:05:18.682 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:05:18.682 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:18.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:18.808+0000 7f3feff57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:18.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:18.816+0000 7f3feff57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:19.011 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:19.011 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:19.024 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-09-22T03:05:18.436347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":256232,"kb_used_data":6824,"kb_used_omap":0,"kb_used_meta":249344,"kb_avail":377231128,"statfs":{"total":386547056640,"available":386284675072,"internally_reserved":0,"allocated":6987776,"data_stored":78836885,"data_compressed":3712245,"data_compressed_allocated":4890624,"data_compressed_original":78004224,"omap_allocated":0,"internal_metadata":255328256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-95223832,"num_objects":-1454,"num_object_clones":0,"num_object_copies":-4362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-59958,"num_write_kb":-1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5094,"num_bytes_recovered":-1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.074573"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":136,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.177832+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:09.177832+0000","last_peered":"2024-09-22T03:05:09.177832+0000","last_clean":"2024-09-22T03:05:09.177832+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:09.177832+0000","last_undegraded":"2024-09-22T03:05:09.177832+0000","last_fullsized":"2024-09-22T03:05:09.177832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":165,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.176362+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:09.176362+0000","last_peered":"2024-09-22T03:05:09.176362+0000","last_clean":"2024-09-22T03:05:09.176362+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:09.176362+0000","last_undegraded":"2024-09-22T03:05:09.176362+0000","last_fullsized":"2024-09-22T03:05:09.176362+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":151,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.183652+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:09.183652+0000","last_peered":"2024-09-22T03:05:09.183652+0000","last_clean":"2024-09-22T03:05:09.183652+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:09.183652+0000","last_undegraded":"2024-09-22T03:05:09.183652+0000","last_fullsized":"2024-09-22T03:05:09.183652+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":146,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.176702+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:09.176702+0000","last_peered":"2024-09-22T03:05:09.176702+0000","last_clean":"2024-09-22T03:05:09.176702+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:09.176702+0000","last_undegraded":"2024-09-22T03:05:09.176702+0000","last_fullsized":"2024-09-22T03:05:09.176702+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":288,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.179432+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:09.179432+0000","last_peered":"2024-09-22T03:05:09.179432+0000","last_clean":"2024-09-22T03:05:09.179432+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:09.179432+0000","last_undegraded":"2024-09-22T03:05:09.179432+0000","last_fullsized":"2024-09-22T03:05:09.179432+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":119,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.177843+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:09.177843+0000","last_peered":"2024-09-22T03:05:09.177843+0000","last_clean":"2024-09-22T03:05:09.177843+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:09.177843+0000","last_undegraded":"2024-09-22T03:05:09.177843+0000","last_fullsized":"2024-09-22T03:05:09.177843+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.180549+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:09.180549+0000","last_peered":"2024-09-22T03:05:09.180549+0000","last_clean":"2024-09-22T03:05:09.180549+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:09.180549+0000","last_undegraded":"2024-09-22T03:05:09.180549+0000","last_fullsized":"2024-09-22T03:05:09.180549+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":126,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.178848+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:09.178848+0000","last_peered":"2024-09-22T03:05:09.178848+0000","last_clean":"2024-09-22T03:05:09.178848+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:09.178848+0000","last_undegraded":"2024-09-22T03:05:09.178848+0000","last_fullsized":"2024-09-22T03:05:09.178848+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":128,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.179852+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:09.179852+0000","last_peered":"2024-09-22T03:05:09.179852+0000","last_clean":"2024-09-22T03:05:09.179852+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:09.179852+0000","last_undegraded":"2024-09-22T03:05:09.179852+0000","last_fullsized":"2024-09-22T03:05:09.179852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691940,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56452,"kb_used_data":1780,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94315388,"statfs":{"total":96636764160,"available":96578957312,"internally_reserved":0,"allocated":1822720,"data_stored":21167006,"data_compressed":995840,"data_compressed_allocated":1310720,"data_compressed_original":20971520,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626502,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64104,"kb_used_data":2328,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94307736,"statfs":{"total":96636764160,"available":96571121664,"internally_reserved":0,"allocated":2383872,"data_stored":29818278,"data_compressed":1408575,"data_compressed_allocated":1855488,"data_compressed_original":29605888,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673354,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75960,"kb_used_data":552,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94295880,"statfs":{"total":96636764160,"available":96558981120,"internally_reserved":0,"allocated":565248,"data_stored":785811,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888243,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":59716,"kb_used_data":2164,"kb_used_omap":0,"kb_used_meta":57536,"kb_avail":94312124,"statfs":{"total":96636764160,"available":96575614976,"internally_reserved":0,"allocated":2215936,"data_stored":27065790,"data_compressed":1277871,"data_compressed_allocated":1683456,"data_compressed_original":26853376,"omap_allocated":0,"internal_metadata":58916864},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:22.026 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:22.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:22.144+0000 7f71882a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:22.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:22.151+0000 7f71882a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:22.328 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:22.328 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:22.340 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-22T03:05:20.363823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":255764,"kb_used_data":2196,"kb_used_omap":0,"kb_used_meta":253504,"kb_avail":377231596,"statfs":{"total":386547056640,"available":386285154304,"internally_reserved":0,"allocated":2248704,"data_stored":2566084,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":259588096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-95223832,"num_objects":-1454,"num_object_clones":0,"num_object_copies":-4362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-59958,"num_write_kb":-1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5094,"num_bytes_recovered":-1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001621"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":142,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.477649+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:18.477649+0000","last_peered":"2024-09-22T03:05:18.477649+0000","last_clean":"2024-09-22T03:05:18.477649+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:18.477649+0000","last_undegraded":"2024-09-22T03:05:18.477649+0000","last_fullsized":"2024-09-22T03:05:18.477649+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":165,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.176362+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:09.176362+0000","last_peered":"2024-09-22T03:05:09.176362+0000","last_clean":"2024-09-22T03:05:09.176362+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:09.176362+0000","last_undegraded":"2024-09-22T03:05:09.176362+0000","last_fullsized":"2024-09-22T03:05:09.176362+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.453926+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:18.453926+0000","last_peered":"2024-09-22T03:05:18.453926+0000","last_clean":"2024-09-22T03:05:18.453926+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:18.453926+0000","last_undegraded":"2024-09-22T03:05:18.453926+0000","last_fullsized":"2024-09-22T03:05:18.453926+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":152,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.462432+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:18.462432+0000","last_peered":"2024-09-22T03:05:18.462432+0000","last_clean":"2024-09-22T03:05:18.462432+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:18.462432+0000","last_undegraded":"2024-09-22T03:05:18.462432+0000","last_fullsized":"2024-09-22T03:05:18.462432+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":288,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.179432+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:09.179432+0000","last_peered":"2024-09-22T03:05:09.179432+0000","last_clean":"2024-09-22T03:05:09.179432+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:09.179432+0000","last_undegraded":"2024-09-22T03:05:09.179432+0000","last_fullsized":"2024-09-22T03:05:09.179432+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":125,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.470344+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:18.470344+0000","last_peered":"2024-09-22T03:05:18.470344+0000","last_clean":"2024-09-22T03:05:18.470344+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:18.470344+0000","last_undegraded":"2024-09-22T03:05:18.470344+0000","last_fullsized":"2024-09-22T03:05:18.470344+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":100,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.453156+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:18.453156+0000","last_peered":"2024-09-22T03:05:18.453156+0000","last_clean":"2024-09-22T03:05:18.453156+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:18.453156+0000","last_undegraded":"2024-09-22T03:05:18.453156+0000","last_fullsized":"2024-09-22T03:05:18.453156+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":132,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.469591+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:18.469591+0000","last_peered":"2024-09-22T03:05:18.469591+0000","last_clean":"2024-09-22T03:05:18.469591+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:18.469591+0000","last_undegraded":"2024-09-22T03:05:18.469591+0000","last_fullsized":"2024-09-22T03:05:18.469591+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":134,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.473342+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:18.473342+0000","last_peered":"2024-09-22T03:05:18.473342+0000","last_clean":"2024-09-22T03:05:18.473342+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:18.473342+0000","last_undegraded":"2024-09-22T03:05:18.473342+0000","last_fullsized":"2024-09-22T03:05:18.473342+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55188,"kb_used_data":516,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94316652,"statfs":{"total":96636764160,"available":96580251648,"internally_reserved":0,"allocated":528384,"data_stored":202025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626507,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62340,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94309500,"statfs":{"total":96636764160,"available":96572928000,"internally_reserved":0,"allocated":577536,"data_stored":789124,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673354,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75960,"kb_used_data":552,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94295880,"statfs":{"total":96636764160,"available":96558981120,"internally_reserved":0,"allocated":565248,"data_stored":785811,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888248,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62276,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94309564,"statfs":{"total":96636764160,"available":96572993536,"internally_reserved":0,"allocated":577536,"data_stored":789124,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:22.340 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-22T03:05:22.340 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:05:22.341 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:22.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:22.440+0000 7ff962263640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:22.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:22.447+0000 7ff962263640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:22.623 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:22.623 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:22.635 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-22T03:05:22.364152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":255764,"kb_used_data":2196,"kb_used_omap":0,"kb_used_meta":253504,"kb_avail":377231596,"statfs":{"total":386547056640,"available":386285154304,"internally_reserved":0,"allocated":2248704,"data_stored":2566084,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":259588096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-47972376,"num_objects":-733,"num_object_clones":0,"num_object_copies":-2199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-60679,"num_write_kb":-1965121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5238,"num_bytes_recovered":-1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001579"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":142,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.477649+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:18.477649+0000","last_peered":"2024-09-22T03:05:18.477649+0000","last_clean":"2024-09-22T03:05:18.477649+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:18.477649+0000","last_undegraded":"2024-09-22T03:05:18.477649+0000","last_fullsized":"2024-09-22T03:05:18.477649+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":165,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.176362+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:09.176362+0000","last_peered":"2024-09-22T03:05:09.176362+0000","last_clean":"2024-09-22T03:05:09.176362+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:09.176362+0000","last_undegraded":"2024-09-22T03:05:09.176362+0000","last_fullsized":"2024-09-22T03:05:09.176362+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.453926+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:18.453926+0000","last_peered":"2024-09-22T03:05:18.453926+0000","last_clean":"2024-09-22T03:05:18.453926+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:18.453926+0000","last_undegraded":"2024-09-22T03:05:18.453926+0000","last_fullsized":"2024-09-22T03:05:18.453926+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":152,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.462432+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:18.462432+0000","last_peered":"2024-09-22T03:05:18.462432+0000","last_clean":"2024-09-22T03:05:18.462432+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:18.462432+0000","last_undegraded":"2024-09-22T03:05:18.462432+0000","last_fullsized":"2024-09-22T03:05:18.462432+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":288,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-22T03:05:09.179432+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:09.179432+0000","last_peered":"2024-09-22T03:05:09.179432+0000","last_clean":"2024-09-22T03:05:09.179432+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:09.179432+0000","last_undegraded":"2024-09-22T03:05:09.179432+0000","last_fullsized":"2024-09-22T03:05:09.179432+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":125,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.470344+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:18.470344+0000","last_peered":"2024-09-22T03:05:18.470344+0000","last_clean":"2024-09-22T03:05:18.470344+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:18.470344+0000","last_undegraded":"2024-09-22T03:05:18.470344+0000","last_fullsized":"2024-09-22T03:05:18.470344+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":100,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.453156+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:18.453156+0000","last_peered":"2024-09-22T03:05:18.453156+0000","last_clean":"2024-09-22T03:05:18.453156+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:18.453156+0000","last_undegraded":"2024-09-22T03:05:18.453156+0000","last_fullsized":"2024-09-22T03:05:18.453156+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":132,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.469591+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:18.469591+0000","last_peered":"2024-09-22T03:05:18.469591+0000","last_clean":"2024-09-22T03:05:18.469591+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:18.469591+0000","last_undegraded":"2024-09-22T03:05:18.469591+0000","last_fullsized":"2024-09-22T03:05:18.469591+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":134,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-22T03:05:18.473342+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:18.473342+0000","last_peered":"2024-09-22T03:05:18.473342+0000","last_clean":"2024-09-22T03:05:18.473342+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:18.473342+0000","last_undegraded":"2024-09-22T03:05:18.473342+0000","last_fullsized":"2024-09-22T03:05:18.473342+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55188,"kb_used_data":516,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94316652,"statfs":{"total":96636764160,"available":96580251648,"internally_reserved":0,"allocated":528384,"data_stored":202025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626507,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62340,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94309500,"statfs":{"total":96636764160,"available":96572928000,"internally_reserved":0,"allocated":577536,"data_stored":789124,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673354,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75960,"kb_used_data":552,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94295880,"statfs":{"total":96636764160,"available":96558981120,"internally_reserved":0,"allocated":565248,"data_stored":785811,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888248,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62276,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94309564,"statfs":{"total":96636764160,"available":96572993536,"internally_reserved":0,"allocated":577536,"data_stored":789124,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:27.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-22T03:05:27.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:27.748+0000 7f7f83997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:27.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:27.753+0000 7f7f83997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:27.768 INFO:teuthology.orchestra.run.smithi081.stdout:{} 2024-09-22T03:05:27.768 INFO:teuthology.orchestra.run.smithi081.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-22T03:05:27.779 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-22T03:05:27.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:27.877+0000 7f7e85463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:27.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:27.886+0000 7f7e85463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:28.111 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.0 to 1 (10000) 2024-09-22T03:05:28.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-22T03:05:28.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:28.227+0000 7f3e50f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:28.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:28.235+0000 7f3e50f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:29.120 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-22T03:05:29.138 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-22T03:05:29.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:29.238+0000 7fd6f7997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:29.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:29.247+0000 7fd6f7997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:30.125 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.1 to 1 (10000) 2024-09-22T03:05:30.143 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-22T03:05:30.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:30.242+0000 7ff285b0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:30.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:30.251+0000 7ff285b0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:31.130 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-22T03:05:31.147 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-22T03:05:31.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:31.246+0000 7faf9eaa9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:31.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:31.255+0000 7faf9eaa9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:32.135 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.3 to 1 (10000) 2024-09-22T03:05:32.154 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-22T03:05:32.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:32.258+0000 7f4dc9f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:32.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:32.267+0000 7f4dc9f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:33.140 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-22T03:05:33.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-22T03:05:33.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:33.263+0000 7f3b9866c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:33.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:33.272+0000 7f3b9866c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:34.110 INFO:teuthology.orchestra.run.smithi081.stderr:reweighted osd.2 to 1 (10000) 2024-09-22T03:05:34.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-22T03:05:34.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:34.229+0000 7f124606c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:34.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:34.238+0000 7f124606c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.156 INFO:teuthology.orchestra.run.smithi081.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-22T03:05:35.175 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-22T03:05:35.175 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T03:05:35.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:05:35.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.275+0000 7f4d2846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.281+0000 7f4d2846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.466 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:35.466 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":71,"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","created":"2024-09-22T02:58:34.078383+0000","modified":"2024-09-22T03:05:35.147224+0000","last_up_change":"2024-09-22T03:04:03.456602+0000","last_in_change":"2024-09-22T02:58:36.162126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T02:58:44.262077+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T02:58:44.315438+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8dc2c56-1fa8-4c94-b1c4-1ff6bb0919b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3135721300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6805","nonce":3135721300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3135721300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3135721300}]},"public_addr":"172.21.15.81:6804/3135721300","cluster_addr":"172.21.15.81:6805/3135721300","heartbeat_back_addr":"172.21.15.81:6807/3135721300","heartbeat_front_addr":"172.21.15.81:6806/3135721300","state":["exists","up"]},{"osd":1,"uuid":"4c95c86d-0d11-4418-921f-c13d799252d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":32,"up_from":34,"up_thru":61,"down_at":33,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":323501905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6803","nonce":323501905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":323501905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":323501905}]},"public_addr":"172.21.15.81:6808/323501905","cluster_addr":"172.21.15.81:6803/323501905","heartbeat_back_addr":"172.21.15.81:6810/323501905","heartbeat_front_addr":"172.21.15.81:6809/323501905","state":["exists","up"]},{"osd":2,"uuid":"c70b3bc0-7a70-4652-a939-acac6432f224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":35,"up_from":42,"up_thru":62,"down_at":38,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":815783814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":815783814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":815783814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":815783814}]},"public_addr":"172.21.15.190:6807/815783814","cluster_addr":"172.21.15.190:6806/815783814","heartbeat_back_addr":"172.21.15.190:6808/815783814","heartbeat_front_addr":"172.21.15.190:6809/815783814","state":["exists","up"]},{"osd":3,"uuid":"4047b1af-c441-4a8f-bc0d-7c2a4f237e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":29,"last_clean_end":35,"up_from":40,"up_thru":54,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2740800897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":2740800897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":2740800897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2740800897}]},"public_addr":"172.21.15.190:6804/2740800897","cluster_addr":"172.21.15.190:6803/2740800897","heartbeat_back_addr":"172.21.15.190:6805/2740800897","heartbeat_front_addr":"172.21.15.190:6802/2740800897","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"2024-09-22T03:02:13.088188+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"2024-09-22T03:03:13.410447+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"2024-09-22T03:03:01.395050+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:05:35.479 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T03:05:35.479 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:05:35.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.577+0000 7f5e4766c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.583+0000 7f5e4766c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.763 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:35.763 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":71,"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","created":"2024-09-22T02:58:34.078383+0000","modified":"2024-09-22T03:05:35.147224+0000","last_up_change":"2024-09-22T03:04:03.456602+0000","last_in_change":"2024-09-22T02:58:36.162126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T02:58:44.262077+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T02:58:44.315438+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8dc2c56-1fa8-4c94-b1c4-1ff6bb0919b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3135721300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6805","nonce":3135721300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3135721300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3135721300}]},"public_addr":"172.21.15.81:6804/3135721300","cluster_addr":"172.21.15.81:6805/3135721300","heartbeat_back_addr":"172.21.15.81:6807/3135721300","heartbeat_front_addr":"172.21.15.81:6806/3135721300","state":["exists","up"]},{"osd":1,"uuid":"4c95c86d-0d11-4418-921f-c13d799252d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":32,"up_from":34,"up_thru":61,"down_at":33,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":323501905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6803","nonce":323501905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":323501905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":323501905}]},"public_addr":"172.21.15.81:6808/323501905","cluster_addr":"172.21.15.81:6803/323501905","heartbeat_back_addr":"172.21.15.81:6810/323501905","heartbeat_front_addr":"172.21.15.81:6809/323501905","state":["exists","up"]},{"osd":2,"uuid":"c70b3bc0-7a70-4652-a939-acac6432f224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":35,"up_from":42,"up_thru":62,"down_at":38,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":815783814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":815783814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":815783814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":815783814}]},"public_addr":"172.21.15.190:6807/815783814","cluster_addr":"172.21.15.190:6806/815783814","heartbeat_back_addr":"172.21.15.190:6808/815783814","heartbeat_front_addr":"172.21.15.190:6809/815783814","state":["exists","up"]},{"osd":3,"uuid":"4047b1af-c441-4a8f-bc0d-7c2a4f237e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":29,"last_clean_end":35,"up_from":40,"up_thru":54,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2740800897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":2740800897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":2740800897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2740800897}]},"public_addr":"172.21.15.190:6804/2740800897","cluster_addr":"172.21.15.190:6803/2740800897","heartbeat_back_addr":"172.21.15.190:6805/2740800897","heartbeat_front_addr":"172.21.15.190:6802/2740800897","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"2024-09-22T03:02:13.088188+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"2024-09-22T03:03:13.410447+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"2024-09-22T03:03:01.395050+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:05:35.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T03:05:35.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T03:05:35.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T03:05:35.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-22T03:05:35.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.884+0000 7fa83246c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.885+0000 7f6bd4797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.884+0000 7f7f82258640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.885+0000 7f142d82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.893+0000 7f6bd4797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.894+0000 7f142d82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.894+0000 7f7f82258640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:35.894+0000 7fa83246c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:35.926 INFO:teuthology.orchestra.run.smithi081.stdout:180388626524 2024-09-22T03:05:35.927 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:05:35.927 INFO:teuthology.orchestra.run.smithi081.stdout:42949673373 2024-09-22T03:05:35.928 INFO:teuthology.orchestra.run.smithi081.stdout:146028888265 2024-09-22T03:05:35.928 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:05:35.929 INFO:teuthology.orchestra.run.smithi081.stdout:171798691961 2024-09-22T03:05:35.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:05:35.930 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-22T03:05:36.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:36.076+0000 7f8bc1157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:36.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:36.076+0000 7f1e7d063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:36.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:36.076+0000 7fd293d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:36.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:36.076+0000 7f3ce14d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:36.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:36.084+0000 7f3ce14d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:36.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:36.084+0000 7f8bc1157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:36.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:36.085+0000 7fd293d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:36.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:36.085+0000 7f1e7d063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:36.276 INFO:teuthology.orchestra.run.smithi081.stdout:146028888263 2024-09-22T03:05:36.287 INFO:teuthology.orchestra.run.smithi081.stdout:171798691955 2024-09-22T03:05:36.293 INFO:tasks.ceph.ceph_manager.ceph:need seq 146028888265 got 146028888263 for osd.1 2024-09-22T03:05:36.298 INFO:teuthology.orchestra.run.smithi081.stdout:180388626522 2024-09-22T03:05:36.299 INFO:teuthology.orchestra.run.smithi081.stdout:42949673369 2024-09-22T03:05:36.300 INFO:tasks.ceph.ceph_manager.ceph:need seq 171798691961 got 171798691955 for osd.3 2024-09-22T03:05:36.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 180388626524 got 180388626522 for osd.2 2024-09-22T03:05:36.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949673373 got 42949673369 for osd.0 2024-09-22T03:05:37.294 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:05:37.301 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-22T03:05:37.314 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:05:37.314 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:05:37.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:37.413+0000 7f08a8458640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:37.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:37.424+0000 7f08a8458640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:37.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:37.452+0000 7f0334997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:37.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:37.455+0000 7f736c757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:37.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:37.460+0000 7f0334997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:37.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:37.457+0000 7fb959858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:37.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:37.464+0000 7f736c757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:37.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:37.473+0000 7fb959858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:37.646 INFO:teuthology.orchestra.run.smithi081.stdout:146028888265 2024-09-22T03:05:37.655 INFO:teuthology.orchestra.run.smithi081.stdout:171798691961 2024-09-22T03:05:37.665 INFO:tasks.ceph.ceph_manager.ceph:need seq 146028888265 got 146028888265 for osd.1 2024-09-22T03:05:37.665 DEBUG:teuthology.parallel:result is None 2024-09-22T03:05:37.669 INFO:teuthology.orchestra.run.smithi081.stdout:42949673373 2024-09-22T03:05:37.673 INFO:tasks.ceph.ceph_manager.ceph:need seq 171798691961 got 171798691961 for osd.3 2024-09-22T03:05:37.673 DEBUG:teuthology.parallel:result is None 2024-09-22T03:05:37.682 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949673373 got 42949673373 for osd.0 2024-09-22T03:05:37.683 DEBUG:teuthology.parallel:result is None 2024-09-22T03:05:37.693 INFO:teuthology.orchestra.run.smithi081.stdout:180388626524 2024-09-22T03:05:37.705 INFO:tasks.ceph.ceph_manager.ceph:need seq 180388626524 got 180388626524 for osd.2 2024-09-22T03:05:37.706 DEBUG:teuthology.parallel:result is None 2024-09-22T03:05:37.706 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-22T03:05:37.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:37.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:37.843+0000 7f131d397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:37.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:37.850+0000 7f131d397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:38.027 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:38.027 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:38.040 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-22T03:05:36.366969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":256064,"kb_used_data":2496,"kb_used_omap":0,"kb_used_meta":253504,"kb_avail":377231296,"statfs":{"total":386547056640,"available":386284847104,"internally_reserved":0,"allocated":2555904,"data_stored":2672689,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":259588096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.231557"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":166,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.166165+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:35.166165+0000","last_peered":"2024-09-22T03:05:35.166165+0000","last_clean":"2024-09-22T03:05:35.166165+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:35.166165+0000","last_undegraded":"2024-09-22T03:05:35.166165+0000","last_fullsized":"2024-09-22T03:05:35.166165+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":195,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163342+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:35.163342+0000","last_peered":"2024-09-22T03:05:35.163342+0000","last_clean":"2024-09-22T03:05:35.163342+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:35.163342+0000","last_undegraded":"2024-09-22T03:05:35.163342+0000","last_fullsized":"2024-09-22T03:05:35.163342+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":181,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165784+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:35.165784+0000","last_peered":"2024-09-22T03:05:35.165784+0000","last_clean":"2024-09-22T03:05:35.165784+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:35.165784+0000","last_undegraded":"2024-09-22T03:05:35.165784+0000","last_fullsized":"2024-09-22T03:05:35.165784+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":176,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165252+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:35.165252+0000","last_peered":"2024-09-22T03:05:35.165252+0000","last_clean":"2024-09-22T03:05:35.165252+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:35.165252+0000","last_undegraded":"2024-09-22T03:05:35.165252+0000","last_fullsized":"2024-09-22T03:05:35.165252+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":318,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165729+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:35.165729+0000","last_peered":"2024-09-22T03:05:35.165729+0000","last_clean":"2024-09-22T03:05:35.165729+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:35.165729+0000","last_undegraded":"2024-09-22T03:05:35.165729+0000","last_fullsized":"2024-09-22T03:05:35.165729+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":149,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165184+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:35.165184+0000","last_peered":"2024-09-22T03:05:35.165184+0000","last_clean":"2024-09-22T03:05:35.165184+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:35.165184+0000","last_undegraded":"2024-09-22T03:05:35.165184+0000","last_fullsized":"2024-09-22T03:05:35.165184+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":124,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163442+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:35.163442+0000","last_peered":"2024-09-22T03:05:35.163442+0000","last_clean":"2024-09-22T03:05:35.163442+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:35.163442+0000","last_undegraded":"2024-09-22T03:05:35.163442+0000","last_fullsized":"2024-09-22T03:05:35.163442+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":156,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.164886+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:35.164886+0000","last_peered":"2024-09-22T03:05:35.164886+0000","last_clean":"2024-09-22T03:05:35.164886+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:35.164886+0000","last_undegraded":"2024-09-22T03:05:35.164886+0000","last_fullsized":"2024-09-22T03:05:35.164886+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":158,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165247+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:35.165247+0000","last_peered":"2024-09-22T03:05:35.165247+0000","last_clean":"2024-09-22T03:05:35.165247+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:35.165247+0000","last_undegraded":"2024-09-22T03:05:35.165247+0000","last_fullsized":"2024-09-22T03:05:35.165247+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55252,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94316588,"statfs":{"total":96636764160,"available":96580186112,"internally_reserved":0,"allocated":593920,"data_stored":225401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626524,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62416,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94309424,"statfs":{"total":96636764160,"available":96572850176,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76044,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94295796,"statfs":{"total":96636764160,"available":96558895104,"internally_reserved":0,"allocated":651264,"data_stored":815750,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62352,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94309488,"statfs":{"total":96636764160,"available":96572915712,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:38.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:38.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:38.138+0000 7facd750e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:38.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:38.144+0000 7facd750e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:38.325 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:38.325 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:38.335 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-22T03:05:36.366969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":256064,"kb_used_data":2496,"kb_used_omap":0,"kb_used_meta":253504,"kb_avail":377231296,"statfs":{"total":386547056640,"available":386284847104,"internally_reserved":0,"allocated":2555904,"data_stored":2672689,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":259588096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.231557"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":166,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.166165+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:35.166165+0000","last_peered":"2024-09-22T03:05:35.166165+0000","last_clean":"2024-09-22T03:05:35.166165+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:35.166165+0000","last_undegraded":"2024-09-22T03:05:35.166165+0000","last_fullsized":"2024-09-22T03:05:35.166165+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":195,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163342+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:35.163342+0000","last_peered":"2024-09-22T03:05:35.163342+0000","last_clean":"2024-09-22T03:05:35.163342+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:35.163342+0000","last_undegraded":"2024-09-22T03:05:35.163342+0000","last_fullsized":"2024-09-22T03:05:35.163342+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":181,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165784+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:35.165784+0000","last_peered":"2024-09-22T03:05:35.165784+0000","last_clean":"2024-09-22T03:05:35.165784+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:35.165784+0000","last_undegraded":"2024-09-22T03:05:35.165784+0000","last_fullsized":"2024-09-22T03:05:35.165784+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":176,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165252+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:35.165252+0000","last_peered":"2024-09-22T03:05:35.165252+0000","last_clean":"2024-09-22T03:05:35.165252+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:35.165252+0000","last_undegraded":"2024-09-22T03:05:35.165252+0000","last_fullsized":"2024-09-22T03:05:35.165252+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":318,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165729+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:35.165729+0000","last_peered":"2024-09-22T03:05:35.165729+0000","last_clean":"2024-09-22T03:05:35.165729+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:35.165729+0000","last_undegraded":"2024-09-22T03:05:35.165729+0000","last_fullsized":"2024-09-22T03:05:35.165729+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":149,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165184+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:35.165184+0000","last_peered":"2024-09-22T03:05:35.165184+0000","last_clean":"2024-09-22T03:05:35.165184+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:35.165184+0000","last_undegraded":"2024-09-22T03:05:35.165184+0000","last_fullsized":"2024-09-22T03:05:35.165184+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":124,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163442+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:35.163442+0000","last_peered":"2024-09-22T03:05:35.163442+0000","last_clean":"2024-09-22T03:05:35.163442+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:35.163442+0000","last_undegraded":"2024-09-22T03:05:35.163442+0000","last_fullsized":"2024-09-22T03:05:35.163442+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":156,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.164886+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:35.164886+0000","last_peered":"2024-09-22T03:05:35.164886+0000","last_clean":"2024-09-22T03:05:35.164886+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:35.164886+0000","last_undegraded":"2024-09-22T03:05:35.164886+0000","last_fullsized":"2024-09-22T03:05:35.164886+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":158,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165247+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:35.165247+0000","last_peered":"2024-09-22T03:05:35.165247+0000","last_clean":"2024-09-22T03:05:35.165247+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:35.165247+0000","last_undegraded":"2024-09-22T03:05:35.165247+0000","last_fullsized":"2024-09-22T03:05:35.165247+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55252,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94316588,"statfs":{"total":96636764160,"available":96580186112,"internally_reserved":0,"allocated":593920,"data_stored":225401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626524,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62416,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94309424,"statfs":{"total":96636764160,"available":96572850176,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76044,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94295796,"statfs":{"total":96636764160,"available":96558895104,"internally_reserved":0,"allocated":651264,"data_stored":815750,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62352,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94309488,"statfs":{"total":96636764160,"available":96572915712,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:38.336 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-22T03:05:38.336 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-22T03:05:38.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:38.436+0000 7fcc6ab97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:38.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:38.445+0000 7fcc6ab97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:38.644 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-22T03:05:38.653 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:05:38.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:38.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:38.751+0000 7ff15c46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:38.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:38.758+0000 7ff15c46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:38.935 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:38.936 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:38.948 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-22T03:05:38.367348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":256064,"kb_used_data":2496,"kb_used_omap":0,"kb_used_meta":253504,"kb_avail":377231296,"statfs":{"total":386547056640,"available":386284847104,"internally_reserved":0,"allocated":2555904,"data_stored":2672689,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":259588096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001274"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":166,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.166165+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:35.166165+0000","last_peered":"2024-09-22T03:05:35.166165+0000","last_clean":"2024-09-22T03:05:35.166165+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:35.166165+0000","last_undegraded":"2024-09-22T03:05:35.166165+0000","last_fullsized":"2024-09-22T03:05:35.166165+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":195,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163342+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:35.163342+0000","last_peered":"2024-09-22T03:05:35.163342+0000","last_clean":"2024-09-22T03:05:35.163342+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:35.163342+0000","last_undegraded":"2024-09-22T03:05:35.163342+0000","last_fullsized":"2024-09-22T03:05:35.163342+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":181,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165784+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:35.165784+0000","last_peered":"2024-09-22T03:05:35.165784+0000","last_clean":"2024-09-22T03:05:35.165784+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:35.165784+0000","last_undegraded":"2024-09-22T03:05:35.165784+0000","last_fullsized":"2024-09-22T03:05:35.165784+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":176,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165252+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:35.165252+0000","last_peered":"2024-09-22T03:05:35.165252+0000","last_clean":"2024-09-22T03:05:35.165252+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:35.165252+0000","last_undegraded":"2024-09-22T03:05:35.165252+0000","last_fullsized":"2024-09-22T03:05:35.165252+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":318,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165729+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:35.165729+0000","last_peered":"2024-09-22T03:05:35.165729+0000","last_clean":"2024-09-22T03:05:35.165729+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:35.165729+0000","last_undegraded":"2024-09-22T03:05:35.165729+0000","last_fullsized":"2024-09-22T03:05:35.165729+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":149,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165184+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:35.165184+0000","last_peered":"2024-09-22T03:05:35.165184+0000","last_clean":"2024-09-22T03:05:35.165184+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:35.165184+0000","last_undegraded":"2024-09-22T03:05:35.165184+0000","last_fullsized":"2024-09-22T03:05:35.165184+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":124,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163442+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:35.163442+0000","last_peered":"2024-09-22T03:05:35.163442+0000","last_clean":"2024-09-22T03:05:35.163442+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:35.163442+0000","last_undegraded":"2024-09-22T03:05:35.163442+0000","last_fullsized":"2024-09-22T03:05:35.163442+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":156,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.164886+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:35.164886+0000","last_peered":"2024-09-22T03:05:35.164886+0000","last_clean":"2024-09-22T03:05:35.164886+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:35.164886+0000","last_undegraded":"2024-09-22T03:05:35.164886+0000","last_fullsized":"2024-09-22T03:05:35.164886+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":158,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165247+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:35.165247+0000","last_peered":"2024-09-22T03:05:35.165247+0000","last_clean":"2024-09-22T03:05:35.165247+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:35.165247+0000","last_undegraded":"2024-09-22T03:05:35.165247+0000","last_fullsized":"2024-09-22T03:05:35.165247+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55252,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94316588,"statfs":{"total":96636764160,"available":96580186112,"internally_reserved":0,"allocated":593920,"data_stored":225401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626524,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62416,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94309424,"statfs":{"total":96636764160,"available":96572850176,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76044,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94295796,"statfs":{"total":96636764160,"available":96558895104,"internally_reserved":0,"allocated":651264,"data_stored":815750,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62352,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94309488,"statfs":{"total":96636764160,"available":96572915712,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:38.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:39.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:39.047+0000 7f68ead57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:39.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:39.053+0000 7f68ead57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:39.239 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:39.240 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:39.252 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-22T03:05:39.115766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":256064,"kb_used_data":2496,"kb_used_omap":0,"kb_used_meta":253504,"kb_avail":377231296,"statfs":{"total":386547056640,"available":386284847104,"internally_reserved":0,"allocated":2555904,"data_stored":2672689,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":259588096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.974993"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":166,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.166165+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:35.166165+0000","last_peered":"2024-09-22T03:05:35.166165+0000","last_clean":"2024-09-22T03:05:35.166165+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:35.166165+0000","last_undegraded":"2024-09-22T03:05:35.166165+0000","last_fullsized":"2024-09-22T03:05:35.166165+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":195,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163342+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:35.163342+0000","last_peered":"2024-09-22T03:05:35.163342+0000","last_clean":"2024-09-22T03:05:35.163342+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:35.163342+0000","last_undegraded":"2024-09-22T03:05:35.163342+0000","last_fullsized":"2024-09-22T03:05:35.163342+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":181,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165784+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:35.165784+0000","last_peered":"2024-09-22T03:05:35.165784+0000","last_clean":"2024-09-22T03:05:35.165784+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:35.165784+0000","last_undegraded":"2024-09-22T03:05:35.165784+0000","last_fullsized":"2024-09-22T03:05:35.165784+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":176,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165252+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:35.165252+0000","last_peered":"2024-09-22T03:05:35.165252+0000","last_clean":"2024-09-22T03:05:35.165252+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:35.165252+0000","last_undegraded":"2024-09-22T03:05:35.165252+0000","last_fullsized":"2024-09-22T03:05:35.165252+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":318,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165729+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:35.165729+0000","last_peered":"2024-09-22T03:05:35.165729+0000","last_clean":"2024-09-22T03:05:35.165729+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:35.165729+0000","last_undegraded":"2024-09-22T03:05:35.165729+0000","last_fullsized":"2024-09-22T03:05:35.165729+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":149,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165184+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:35.165184+0000","last_peered":"2024-09-22T03:05:35.165184+0000","last_clean":"2024-09-22T03:05:35.165184+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:35.165184+0000","last_undegraded":"2024-09-22T03:05:35.165184+0000","last_fullsized":"2024-09-22T03:05:35.165184+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":124,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163442+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:35.163442+0000","last_peered":"2024-09-22T03:05:35.163442+0000","last_clean":"2024-09-22T03:05:35.163442+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:35.163442+0000","last_undegraded":"2024-09-22T03:05:35.163442+0000","last_fullsized":"2024-09-22T03:05:35.163442+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":156,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.164886+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:35.164886+0000","last_peered":"2024-09-22T03:05:35.164886+0000","last_clean":"2024-09-22T03:05:35.164886+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:35.164886+0000","last_undegraded":"2024-09-22T03:05:35.164886+0000","last_fullsized":"2024-09-22T03:05:35.164886+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":158,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165247+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:35.165247+0000","last_peered":"2024-09-22T03:05:35.165247+0000","last_clean":"2024-09-22T03:05:35.165247+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:35.165247+0000","last_undegraded":"2024-09-22T03:05:35.165247+0000","last_fullsized":"2024-09-22T03:05:35.165247+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55252,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94316588,"statfs":{"total":96636764160,"available":96580186112,"internally_reserved":0,"allocated":593920,"data_stored":225401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626527,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62416,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94309424,"statfs":{"total":96636764160,"available":96572850176,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76044,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94295796,"statfs":{"total":96636764160,"available":96558895104,"internally_reserved":0,"allocated":651264,"data_stored":815750,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62352,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94309488,"statfs":{"total":96636764160,"available":96572915712,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:39.253 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:05:39.253 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:39.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:39.353+0000 7f399f82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:39.358+0000 7f399f82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:39.536 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:39.537 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:39.549 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-22T03:05:39.115766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":256064,"kb_used_data":2496,"kb_used_omap":0,"kb_used_meta":253504,"kb_avail":377231296,"statfs":{"total":386547056640,"available":386284847104,"internally_reserved":0,"allocated":2555904,"data_stored":2672689,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":259588096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.974993"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":166,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.166165+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:35.166165+0000","last_peered":"2024-09-22T03:05:35.166165+0000","last_clean":"2024-09-22T03:05:35.166165+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:35.166165+0000","last_undegraded":"2024-09-22T03:05:35.166165+0000","last_fullsized":"2024-09-22T03:05:35.166165+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":195,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163342+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:35.163342+0000","last_peered":"2024-09-22T03:05:35.163342+0000","last_clean":"2024-09-22T03:05:35.163342+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:35.163342+0000","last_undegraded":"2024-09-22T03:05:35.163342+0000","last_fullsized":"2024-09-22T03:05:35.163342+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":181,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165784+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:35.165784+0000","last_peered":"2024-09-22T03:05:35.165784+0000","last_clean":"2024-09-22T03:05:35.165784+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:35.165784+0000","last_undegraded":"2024-09-22T03:05:35.165784+0000","last_fullsized":"2024-09-22T03:05:35.165784+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":176,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165252+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:35.165252+0000","last_peered":"2024-09-22T03:05:35.165252+0000","last_clean":"2024-09-22T03:05:35.165252+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:35.165252+0000","last_undegraded":"2024-09-22T03:05:35.165252+0000","last_fullsized":"2024-09-22T03:05:35.165252+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":318,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165729+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:35.165729+0000","last_peered":"2024-09-22T03:05:35.165729+0000","last_clean":"2024-09-22T03:05:35.165729+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:35.165729+0000","last_undegraded":"2024-09-22T03:05:35.165729+0000","last_fullsized":"2024-09-22T03:05:35.165729+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":149,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165184+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:35.165184+0000","last_peered":"2024-09-22T03:05:35.165184+0000","last_clean":"2024-09-22T03:05:35.165184+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:35.165184+0000","last_undegraded":"2024-09-22T03:05:35.165184+0000","last_fullsized":"2024-09-22T03:05:35.165184+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":124,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163442+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:35.163442+0000","last_peered":"2024-09-22T03:05:35.163442+0000","last_clean":"2024-09-22T03:05:35.163442+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:35.163442+0000","last_undegraded":"2024-09-22T03:05:35.163442+0000","last_fullsized":"2024-09-22T03:05:35.163442+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":156,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.164886+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:35.164886+0000","last_peered":"2024-09-22T03:05:35.164886+0000","last_clean":"2024-09-22T03:05:35.164886+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:35.164886+0000","last_undegraded":"2024-09-22T03:05:35.164886+0000","last_fullsized":"2024-09-22T03:05:35.164886+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":158,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165247+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:35.165247+0000","last_peered":"2024-09-22T03:05:35.165247+0000","last_clean":"2024-09-22T03:05:35.165247+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:35.165247+0000","last_undegraded":"2024-09-22T03:05:35.165247+0000","last_fullsized":"2024-09-22T03:05:35.165247+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55252,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94316588,"statfs":{"total":96636764160,"available":96580186112,"internally_reserved":0,"allocated":593920,"data_stored":225401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626527,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62416,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94309424,"statfs":{"total":96636764160,"available":96572850176,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76044,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94295796,"statfs":{"total":96636764160,"available":96558895104,"internally_reserved":0,"allocated":651264,"data_stored":815750,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62352,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94309488,"statfs":{"total":96636764160,"available":96572915712,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:39.550 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:05:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:39.649+0000 7f337126c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:39.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:39.654+0000 7f337126c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:39.833 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:39.833 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":72,"fsid":"ba6605a2-3158-4c2e-9364-faeb0f665984","created":"2024-09-22T02:58:34.078383+0000","modified":"2024-09-22T03:05:39.109772+0000","last_up_change":"2024-09-22T03:04:03.456602+0000","last_in_change":"2024-09-22T02:58:36.162126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T02:58:44.262077+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T02:58:44.315438+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8dc2c56-1fa8-4c94-b1c4-1ff6bb0919b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3135721300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6805","nonce":3135721300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3135721300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3135721300}]},"public_addr":"172.21.15.81:6804/3135721300","cluster_addr":"172.21.15.81:6805/3135721300","heartbeat_back_addr":"172.21.15.81:6807/3135721300","heartbeat_front_addr":"172.21.15.81:6806/3135721300","state":["exists","up"]},{"osd":1,"uuid":"4c95c86d-0d11-4418-921f-c13d799252d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":32,"up_from":34,"up_thru":61,"down_at":33,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":323501905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6803","nonce":323501905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":323501905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":323501905}]},"public_addr":"172.21.15.81:6808/323501905","cluster_addr":"172.21.15.81:6803/323501905","heartbeat_back_addr":"172.21.15.81:6810/323501905","heartbeat_front_addr":"172.21.15.81:6809/323501905","state":["exists","up"]},{"osd":2,"uuid":"c70b3bc0-7a70-4652-a939-acac6432f224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":35,"up_from":42,"up_thru":62,"down_at":38,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":815783814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":815783814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":815783814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":815783814}]},"public_addr":"172.21.15.190:6807/815783814","cluster_addr":"172.21.15.190:6806/815783814","heartbeat_back_addr":"172.21.15.190:6808/815783814","heartbeat_front_addr":"172.21.15.190:6809/815783814","state":["exists","up"]},{"osd":3,"uuid":"4047b1af-c441-4a8f-bc0d-7c2a4f237e18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":29,"last_clean_end":35,"up_from":40,"up_thru":54,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2740800897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":2740800897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":2740800897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2740800897}]},"public_addr":"172.21.15.190:6804/2740800897","cluster_addr":"172.21.15.190:6803/2740800897","heartbeat_back_addr":"172.21.15.190:6805/2740800897","heartbeat_front_addr":"172.21.15.190:6802/2740800897","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"2024-09-22T03:02:13.088188+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"2024-09-22T03:03:13.410447+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"2024-09-22T03:03:01.395050+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:05:40.848 INFO:tasks.ceph:Scrubbing osd.0 2024-09-22T03:05:40.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:05:40.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:40.960+0000 7fbab8ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:40.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:40.971+0000 7fbab8ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:40.984 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-22T03:05:40.984 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "" 2024-09-22T03:05:40.984 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-22T03:05:40.995 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-22T03:05:41.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:41.094+0000 7fdaf94d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:41.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:41.102+0000 7fdaf94d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:41.285 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 0 to deep-scrub 2024-09-22T03:05:41.301 INFO:tasks.ceph:Scrubbing osd.1 2024-09-22T03:05:41.301 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:05:41.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:41.406+0000 7f49eff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:41.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:41.417+0000 7f49eff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:41.429 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-22T03:05:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "" 2024-09-22T03:05:41.429 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-22T03:05:41.440 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-22T03:05:41.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:41.539+0000 7f57b3eb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:41.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:41.546+0000 7f57b3eb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:41.724 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 1 to deep-scrub 2024-09-22T03:05:41.740 INFO:tasks.ceph:Scrubbing osd.2 2024-09-22T03:05:41.740 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:05:41.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:41.840+0000 7fa50aeb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:41.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:41.851+0000 7fa50aeb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:41.864 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-22T03:05:41.864 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "" 2024-09-22T03:05:41.864 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-22T03:05:41.875 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-22T03:05:41.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:41.973+0000 7f900b463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:41.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:41.981+0000 7f900b463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:42.157 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 2 to deep-scrub 2024-09-22T03:05:42.173 INFO:tasks.ceph:Scrubbing osd.3 2024-09-22T03:05:42.173 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:05:42.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:42.272+0000 7fc69b06c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:42.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:42.283+0000 7fc69b06c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:42.297 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-09-22T03:05:42.297 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "" 2024-09-22T03:05:42.297 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-09-22T03:05:42.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-22T03:05:42.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:42.407+0000 7fe84e317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:42.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:42.414+0000 7fe84e317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:42.594 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 3 to deep-scrub 2024-09-22T03:05:42.608 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:05:42.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:42.708+0000 7f73d5397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:42.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:05:42.715+0000 7f73d5397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:05:42.893 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:05:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:05:42.906 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-22T03:05:42.368154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":256072,"kb_used_data":2504,"kb_used_omap":0,"kb_used_meta":253504,"kb_avail":377231288,"statfs":{"total":386547056640,"available":386284838912,"internally_reserved":0,"allocated":2564096,"data_stored":2675605,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":259588096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001564"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":169,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-22T03:05:39.124022+0000","last_change":"2024-09-22T03:04:05.279414+0000","last_active":"2024-09-22T03:05:39.124022+0000","last_peered":"2024-09-22T03:05:39.124022+0000","last_clean":"2024-09-22T03:05:39.124022+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:39.124022+0000","last_undegraded":"2024-09-22T03:05:39.124022+0000","last_fullsized":"2024-09-22T03:05:39.124022+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":195,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163342+0000","last_change":"2024-09-22T03:04:05.275277+0000","last_active":"2024-09-22T03:05:35.163342+0000","last_peered":"2024-09-22T03:05:35.163342+0000","last_clean":"2024-09-22T03:05:35.163342+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:35.163342+0000","last_undegraded":"2024-09-22T03:05:35.163342+0000","last_fullsized":"2024-09-22T03:05:35.163342+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":181,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165784+0000","last_change":"2024-09-22T03:04:05.121802+0000","last_active":"2024-09-22T03:05:35.165784+0000","last_peered":"2024-09-22T03:05:35.165784+0000","last_clean":"2024-09-22T03:05:35.165784+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:35.165784+0000","last_undegraded":"2024-09-22T03:05:35.165784+0000","last_fullsized":"2024-09-22T03:05:35.165784+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":179,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-22T03:05:39.123606+0000","last_change":"2024-09-22T03:04:03.530708+0000","last_active":"2024-09-22T03:05:39.123606+0000","last_peered":"2024-09-22T03:05:39.123606+0000","last_clean":"2024-09-22T03:05:39.123606+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:39.123606+0000","last_undegraded":"2024-09-22T03:05:39.123606+0000","last_fullsized":"2024-09-22T03:05:39.123606+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":318,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165729+0000","last_change":"2024-09-22T03:04:04.203892+0000","last_active":"2024-09-22T03:05:35.165729+0000","last_peered":"2024-09-22T03:05:35.165729+0000","last_clean":"2024-09-22T03:05:35.165729+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:35.165729+0000","last_undegraded":"2024-09-22T03:05:35.165729+0000","last_fullsized":"2024-09-22T03:05:35.165729+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":149,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165184+0000","last_change":"2024-09-22T03:04:05.203402+0000","last_active":"2024-09-22T03:05:35.165184+0000","last_peered":"2024-09-22T03:05:35.165184+0000","last_clean":"2024-09-22T03:05:35.165184+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:35.165184+0000","last_undegraded":"2024-09-22T03:05:35.165184+0000","last_fullsized":"2024-09-22T03:05:35.165184+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":124,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.163442+0000","last_change":"2024-09-22T03:04:05.127664+0000","last_active":"2024-09-22T03:05:35.163442+0000","last_peered":"2024-09-22T03:05:35.163442+0000","last_clean":"2024-09-22T03:05:35.163442+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:35.163442+0000","last_undegraded":"2024-09-22T03:05:35.163442+0000","last_fullsized":"2024-09-22T03:05:35.163442+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":156,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.164886+0000","last_change":"2024-09-22T03:04:05.369513+0000","last_active":"2024-09-22T03:05:35.164886+0000","last_peered":"2024-09-22T03:05:35.164886+0000","last_clean":"2024-09-22T03:05:35.164886+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:35.164886+0000","last_undegraded":"2024-09-22T03:05:35.164886+0000","last_fullsized":"2024-09-22T03:05:35.164886+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":158,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-22T03:05:35.165247+0000","last_change":"2024-09-22T03:04:03.512928+0000","last_active":"2024-09-22T03:05:35.165247+0000","last_peered":"2024-09-22T03:05:35.165247+0000","last_clean":"2024-09-22T03:05:35.165247+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:35.165247+0000","last_undegraded":"2024-09-22T03:05:35.165247+0000","last_fullsized":"2024-09-22T03:05:35.165247+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T02:58:44.315779+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55260,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94316580,"statfs":{"total":96636764160,"available":96580177920,"internally_reserved":0,"allocated":602112,"data_stored":228317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626527,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62416,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94309424,"statfs":{"total":96636764160,"available":96572850176,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76044,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94295796,"statfs":{"total":96636764160,"available":96558895104,"internally_reserved":0,"allocated":651264,"data_stored":815750,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62352,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94309488,"statfs":{"total":96636764160,"available":96572915712,"internally_reserved":0,"allocated":655360,"data_stored":815769,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:05:42.911 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-22T02:58:44.315779+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=2, tm_min=58, tm_sec=44, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=5, tm_sec=39, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:05:42.911 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-22T02:58:44.315779+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=2, tm_min=58, tm_sec=44, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=5, tm_sec=39, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:05:42.911 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-22T02:58:44.315779+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=2, tm_min=58, tm_sec=44, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=5, tm_sec=39, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:05:42.911 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-22T02:58:44.315779+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=2, tm_min=58, tm_sec=44, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=5, tm_sec=39, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:05:42.912 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-22T02:58:44.315779+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=2, tm_min=58, tm_sec=44, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=5, tm_sec=39, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:05:42.912 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-22T02:58:44.315779+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=2, tm_min=58, tm_sec=44, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=5, tm_sec=39, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:05:42.912 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-22T02:58:44.315779+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=2, tm_min=58, tm_sec=44, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=5, tm_sec=39, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:05:42.912 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-22T02:58:44.315779+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=2, tm_min=58, tm_sec=44, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=5, tm_sec=39, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:05:42.912 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-22T02:58:44.315779+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=2, tm_min=58, tm_sec=44, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=5, tm_sec=39, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:05:42.912 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-22T03:06:02.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:06:03.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:06:03.026+0000 7fb27f6ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:06:03.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:06:03.033+0000 7fb27f6ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:06:03.208 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-09-22T03:06:03.209 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-09-22T03:06:03.221 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-22T03:06:02.372501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":256096,"kb_used_data":2528,"kb_used_omap":0,"kb_used_meta":253504,"kb_avail":377231264,"statfs":{"total":386547056640,"available":386284814336,"internally_reserved":0,"allocated":2588672,"data_stored":2684353,"data_compressed":89877,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":259588096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002660"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":171,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-22T03:05:42.595184+0000","last_change":"2024-09-22T03:05:42.601321+0000","last_active":"2024-09-22T03:05:42.595184+0000","last_peered":"2024-09-22T03:05:42.595184+0000","last_clean":"2024-09-22T03:05:42.595184+0000","last_became_active":"2024-09-22T03:04:04.175712+0000","last_became_peered":"2024-09-22T03:04:04.175712+0000","last_unstale":"2024-09-22T03:05:42.595184+0000","last_undegraded":"2024-09-22T03:05:42.595184+0000","last_fullsized":"2024-09-22T03:05:42.595184+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:05:42.601164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:05:42.601164+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":200,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-22T03:05:41.285356+0000","last_change":"2024-09-22T03:05:41.290991+0000","last_active":"2024-09-22T03:05:41.285356+0000","last_peered":"2024-09-22T03:05:41.285356+0000","last_clean":"2024-09-22T03:05:41.285356+0000","last_became_active":"2024-09-22T03:04:04.200459+0000","last_became_peered":"2024-09-22T03:04:04.200459+0000","last_unstale":"2024-09-22T03:05:41.285356+0000","last_undegraded":"2024-09-22T03:05:41.285356+0000","last_fullsized":"2024-09-22T03:05:41.285356+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:05:41.290821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:05:41.290821+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":186,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-22T03:05:42.158681+0000","last_change":"2024-09-22T03:05:42.167837+0000","last_active":"2024-09-22T03:05:42.158681+0000","last_peered":"2024-09-22T03:05:42.158681+0000","last_clean":"2024-09-22T03:05:42.158681+0000","last_became_active":"2024-09-22T03:04:04.243420+0000","last_became_peered":"2024-09-22T03:04:04.243420+0000","last_unstale":"2024-09-22T03:05:42.158681+0000","last_undegraded":"2024-09-22T03:05:42.158681+0000","last_fullsized":"2024-09-22T03:05:42.158681+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'2","last_scrub_stamp":"2024-09-22T03:05:42.167698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:05:42.167698+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":181,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-22T03:05:42.595706+0000","last_change":"2024-09-22T03:05:42.726740+0000","last_active":"2024-09-22T03:05:42.595706+0000","last_peered":"2024-09-22T03:05:42.595706+0000","last_clean":"2024-09-22T03:05:42.595706+0000","last_became_active":"2024-09-22T03:03:35.296500+0000","last_became_peered":"2024-09-22T03:03:35.296500+0000","last_unstale":"2024-09-22T03:05:42.595706+0000","last_undegraded":"2024-09-22T03:05:42.595706+0000","last_fullsized":"2024-09-22T03:05:42.595706+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'1","last_scrub_stamp":"2024-09-22T03:05:42.726622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:05:42.726622+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":323,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-22T03:05:41.286038+0000","last_change":"2024-09-22T03:05:41.381784+0000","last_active":"2024-09-22T03:05:41.286038+0000","last_peered":"2024-09-22T03:05:41.286038+0000","last_clean":"2024-09-22T03:05:41.286038+0000","last_became_active":"2024-09-22T03:04:04.200613+0000","last_became_peered":"2024-09-22T03:04:04.200613+0000","last_unstale":"2024-09-22T03:05:41.286038+0000","last_undegraded":"2024-09-22T03:05:41.286038+0000","last_fullsized":"2024-09-22T03:05:41.286038+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'97","last_scrub_stamp":"2024-09-22T03:05:41.381627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:05:41.381627+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":154,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-22T03:05:41.725463+0000","last_change":"2024-09-22T03:05:41.769403+0000","last_active":"2024-09-22T03:05:41.725463+0000","last_peered":"2024-09-22T03:05:41.725463+0000","last_clean":"2024-09-22T03:05:41.725463+0000","last_became_active":"2024-09-22T03:04:04.188420+0000","last_became_peered":"2024-09-22T03:04:04.188420+0000","last_unstale":"2024-09-22T03:05:41.725463+0000","last_undegraded":"2024-09-22T03:05:41.725463+0000","last_fullsized":"2024-09-22T03:05:41.725463+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:05:41.769324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:05:41.769324+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":129,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-22T03:05:42.157868+0000","last_change":"2024-09-22T03:05:42.164674+0000","last_active":"2024-09-22T03:05:42.157868+0000","last_peered":"2024-09-22T03:05:42.157868+0000","last_clean":"2024-09-22T03:05:42.157868+0000","last_became_active":"2024-09-22T03:04:04.182951+0000","last_became_peered":"2024-09-22T03:04:04.182951+0000","last_unstale":"2024-09-22T03:05:42.157868+0000","last_undegraded":"2024-09-22T03:05:42.157868+0000","last_fullsized":"2024-09-22T03:05:42.157868+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:05:42.164564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:05:42.164564+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":161,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-22T03:05:41.725378+0000","last_change":"2024-09-22T03:05:41.768232+0000","last_active":"2024-09-22T03:05:41.725378+0000","last_peered":"2024-09-22T03:05:41.725378+0000","last_clean":"2024-09-22T03:05:41.725378+0000","last_became_active":"2024-09-22T03:04:04.300976+0000","last_became_peered":"2024-09-22T03:04:04.300976+0000","last_unstale":"2024-09-22T03:05:41.725378+0000","last_undegraded":"2024-09-22T03:05:41.725378+0000","last_fullsized":"2024-09-22T03:05:41.725378+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:05:41.768040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:05:41.768040+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":163,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-22T03:05:41.725897+0000","last_change":"2024-09-22T03:05:41.793140+0000","last_active":"2024-09-22T03:05:41.725897+0000","last_peered":"2024-09-22T03:05:41.725897+0000","last_clean":"2024-09-22T03:05:41.725897+0000","last_became_active":"2024-09-22T03:03:35.295528+0000","last_became_peered":"2024-09-22T03:03:35.295528+0000","last_unstale":"2024-09-22T03:05:41.725897+0000","last_undegraded":"2024-09-22T03:05:41.725897+0000","last_fullsized":"2024-09-22T03:05:41.725897+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:05:41.792969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:05:41.792969+0000","last_clean_scrub_stamp":"2024-09-22T02:58:44.315779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":40,"seq":171798691985,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55260,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":94316580,"statfs":{"total":96636764160,"available":96580177920,"internally_reserved":0,"allocated":602112,"data_stored":228317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":42,"seq":180388626547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62424,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":94309416,"statfs":{"total":96636764160,"available":96572841984,"internally_reserved":0,"allocated":663552,"data_stored":818685,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673399,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76052,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94295788,"statfs":{"total":96636764160,"available":96558886912,"internally_reserved":0,"allocated":659456,"data_stored":818666,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":34,"seq":146028888288,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62360,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":61696,"kb_avail":94309480,"statfs":{"total":96636764160,"available":96572907520,"internally_reserved":0,"allocated":663552,"data_stored":818685,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":63176704},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:06:03.222 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-22T03:06:03.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:06:03.318+0000 7fd24b557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:06:03.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-09-22T03:06:03.328+0000 7fd24b557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:06:03.534 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-22T03:06:03.534 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-22T03:06:03.535 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T03:06:03.535 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:06:08.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-22T03:06:09.636 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T03:06:09.637 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:06:09.637 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:06:14.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:06:14.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-22T03:06:15.738 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:06:15.739 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:06:15.739 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:06:19.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:06:19.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:06:21.841 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:06:21.841 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:06:21.841 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:06:24.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-22T03:06:24.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-22T03:06:27.943 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:06:27.943 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-22T03:06:27.944 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-22T03:06:27.944 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:06:27.978 INFO:tasks.ceph.mgr.x:Stopped 2024-09-22T03:06:27.978 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-22T03:06:27.978 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-22T03:06:27.978 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:06:27.992 INFO:tasks.ceph.mon.a:Stopped 2024-09-22T03:06:27.992 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-22T03:06:27.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-22T03:06:28.074 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi081.front.sepia.ceph.com 2024-09-22T03:06:28.074 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-22T03:06:28.194 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi081.front.sepia.ceph.com 2024-09-22T03:06:28.194 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-22T03:06:28.263 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-22T03:06:28.264 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-22T03:06:28.388 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-22T03:06:28.388 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-22T03:06:28.459 INFO:tasks.ceph:Archiving mon data... 2024-09-22T03:06:28.460 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914712/data/mon.a.tgz 2024-09-22T03:06:28.461 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-09-22T03:06:28.479 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.ZqKkKcclWT 2024-09-22T03:06:28.479 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ZqKkKcclWT 2024-09-22T03:06:28.621 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.ZqKkKcclWT 2024-09-22T03:06:28.705 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.ZqKkKcclWT is 314KB 2024-09-22T03:06:28.783 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.ZqKkKcclWT 2024-09-22T03:06:28.803 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-22T03:06:28.804 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T03:06:28.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T03:06:28.930 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T03:06:28.949 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T03:06:28.949 INFO:tasks.ceph:Archiving crash dumps... 2024-09-22T03:06:28.952 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914712/remote/smithi081/crash 2024-09-22T03:06:28.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T03:06:28.996 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914712/remote/smithi190/crash 2024-09-22T03:06:28.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T03:06:29.053 INFO:tasks.ceph:Compressing logs... 2024-09-22T03:06:29.053 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T03:06:29.056 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T03:06:29.077 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-22T03:06:29.077 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30884.log 2024-09-22T03:06:29.088 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28154.log 2024-09-22T03:06:29.089 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28205.log 2024-09-22T03:06:29.089 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28285.log 2024-09-22T03:06:29.089 INFO:teuthology.orchestra.run.smithi190.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28154.log.gz 2024-09-22T03:06:29.089 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28205.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-22T03:06:29.089 INFO:teuthology.orchestra.run.smithi190.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28205.log.gz 2024-09-22T03:06:29.089 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28285.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-22T03:06:29.089 INFO:teuthology.orchestra.run.smithi190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28285.log.gz 2024-09-22T03:06:29.089 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28245.log 2024-09-22T03:06:29.089 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-22T03:06:29.089 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28245.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28245.log.gz 2024-09-22T03:06:29.142 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.3.log: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-22T03:06:29.181 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38897.log 2024-09-22T03:06:29.181 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.30884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28421.log 2024-09-22T03:06:29.181 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30884.log.gz 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38050.log 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38897.log.gz 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31541.log 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28421.log.gz 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36678.log 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38050.log.gz 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37693.log 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31541.log.gz 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40390.log 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36678.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36678.log.gz 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36907.log 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37693.log.gz 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40390.log: /var/log/ceph/ceph-client.admin.38099.log 2024-09-22T03:06:29.182 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40390.log.gz 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33108.log 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36907.log.gz 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29338.log 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38099.log.gz 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39834.log 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33108.log.gz 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37565.log 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29338.log.gz 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36820.log 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39834.log.gz 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37125.log 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37565.log.gz 2024-09-22T03:06:29.183 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31143.log 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36820.log.gz 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29254.log 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37125.log.gz 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.30037.log 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31143.log.gz 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38225.log 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29254.log.gz 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.30037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40620.log 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.0.30037.log.gz 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40010.log 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38225.log.gz 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40620.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.40620.log.gz -5 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39170.log 2024-09-22T03:06:29.184 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40010.log.gz 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29880.log 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39170.log.gz 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.41065.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34680.log 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29880.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29880.log.gz 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37434.log 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33764.log 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34680.log.gz 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30408.log 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37434.log.gz 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.33764.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29105.log 2024-09-22T03:06:29.185 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33764.log.gz 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35345.log 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30408.log.gz 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34771.log 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29105.log.gz 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29750.log 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35345.log.gz 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41234.log 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34771.log.gz 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37165.log 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.29750.log.gz 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39394.log 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41234.log.gz 2024-09-22T03:06:29.186 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33626.log 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37165.log.gz 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39394.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39394.log.gz 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32990.log 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33626.log.gz 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38449.log 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33408.log 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32990.log.gz 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35609.log 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38449.log.gz 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33408.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33408.log.gz 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29106.log 2024-09-22T03:06:29.187 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39345.log 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35609.log.gz 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29106.log: gzip -5 --verbose 50.0% -- -- replaced with /var/log/ceph/ceph-client.admin.29106.log.gz /var/log/ceph/ceph-client.admin.32695.log 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37476.log 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39345.log.gz 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30007.log 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32695.log.gz 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29751.log 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37476.log.gz 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34113.log 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30007.log.gz 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39478.log 2024-09-22T03:06:29.188 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29751.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29751.log.gz 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33586.log 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34113.log.gz 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39478.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39478.log.gz 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32744.log 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35563.log 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33586.log.gz 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29548.log 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32744.log.gz 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29296.log 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35563.log.gz 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37036.log 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29548.log.gz 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31332.log 2024-09-22T03:06:29.189 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29296.log.gz 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37036.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37036.log.gz 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29145.log 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41319.log 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31332.log: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.31332.log.gz 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31679.log 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29145.log.gz 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41361.log 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41319.log.gz 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32254.log 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31679.log.gz 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38407.log 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41361.log.gz 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35979.log 2024-09-22T03:06:29.190 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32254.log.gz 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31192.log 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38407.log.gz 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38946.log 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35979.log.gz 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40789.log 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31192.log.gz 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38946.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38946.log.gz 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33248.log 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37343.log 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28159.log 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33248.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33248.log.gz 2024-09-22T03:06:29.191 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32555.log 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37343.log.gz 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39121.log 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28159.log.gz 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28977.log 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.32555.log.gz 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33986.log 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39121.log.gz 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28394.log 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28977.log.gz 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35218.log 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33986.log.gz 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30508.log 2024-09-22T03:06:29.192 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28394.log.gz 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40221.log 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35218.log.gz 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30508.log: gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30508.log.gz -5 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39028.log 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34028.log 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40221.log.gz 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35040.log 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39028.log.gz 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35471.log 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34028.log.gz 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35040.log: gzip -5 49.6% --verbose -- replaced with /var/log/ceph/ceph-client.admin.35040.log.gz -- 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.37651.log 2024-09-22T03:06:29.193 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37826.log 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35471.log.gz 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34820.log 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37651.log.gz 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35655.log 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37826.log.gz 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34820.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34820.log.gz 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40618.log 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34334.log 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35655.log.gz 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31501.log 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40618.log.gz 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39075.log 2024-09-22T03:06:29.194 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34334.log.gz 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31808.log 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31501.log.gz 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36209.log 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39075.log.gz 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36303.log 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31808.log.gz 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30973.log 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36209.log.gz 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39702.log 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36303.log.gz 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35839.log 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30973.log.gz 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37245.log 2024-09-22T03:06:29.195 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39702.log.gz 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35839.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35839.log.gz 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34068.log 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37245.log.gz 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40897.log 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34068.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34068.log.gz 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35265.log 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39212.log 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40897.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40897.log.gz 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35265.log: 87.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35265.log.gz 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31897.log 2024-09-22T03:06:29.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36073.log 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39212.log.gz 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36491.log 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31897.log.gz 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31848.log 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36073.log.gz 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34960.log 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36491.log.gz 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28650.log 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31848.log.gz 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35100.log 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34960.log.gz 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41107.log 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28650.log.gz 2024-09-22T03:06:29.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35000.log 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35100.log.gz 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37733.log 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41107.log.gz 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32162.log 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35000.log.gz 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32646.log 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37733.log.gz 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39793.log 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32162.log.gz 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38855.log 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32646.log.gz 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38183.log 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39793.log.gz 2024-09-22T03:06:29.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29590.log 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38855.log.gz 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.38183.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31421.log 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38183.log.gz 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.29590.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39744.log 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29590.log.gz 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37959.log 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31421.log.gz 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36778.log 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39744.log.gz 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37959.log: gzip 49.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.37959.log.gz -- 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.30933.log 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30715.log 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36778.log.gz 2024-09-22T03:06:29.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30448.log 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30933.log.gz 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41529.log 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30715.log.gz 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30448.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.31093.log -- replaced with /var/log/ceph/ceph-client.admin.30448.log.gz 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34631.log 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41529.log.gz 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36116.log 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31093.log.gz 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33157.log 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34631.log.gz 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40094.log 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36116.log.gz 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33157.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33157.log.gz 2024-09-22T03:06:29.200 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36256.log 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40957.log 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40094.log.gz 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35385.log 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36256.log.gz 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37780.log 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33546.log 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.35385.log.gz 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36026.log 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37780.log.gz 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35169.log 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33546.log.gz 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36540.log 2024-09-22T03:06:29.201 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36026.log.gz 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36589.log 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35169.log.gz 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37979.log 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36540.log.gz 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31768.log 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36589.log.gz 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34203.log 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37979.log.gz 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34862.log 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31768.log.gz 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38274.log 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34203.log.gz 2024-09-22T03:06:29.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39876.log 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34862.log.gz 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37294.log 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38274.log.gz 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39610.log 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39876.log.gz 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40621.log 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37294.log.gz 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36947.log 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39610.log.gz 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33457.log 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40621.log.gz 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32336.log 2024-09-22T03:06:29.203 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36947.log.gz 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37917.log 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33457.log.gz 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33059.log 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32336.log.gz 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31283.log 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37917.log.gz 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39660.log 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33059.log.gz 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31699.log 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31283.log.gz 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30795.log 2024-09-22T03:06:29.204 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39660.log.gz 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36162.log 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31699.log.gz 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28975.log 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30795.log.gz 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40787.log 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36162.log.gz 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36444.log 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28975.log.gz 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31239.log 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40787.log.gz 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40619.log 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36444.log.gz 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41739.log 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31239.log.gz 2024-09-22T03:06:29.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40619.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27083.log 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40619.log.gz 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33666.log 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41739.log.gz 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.tmp-client.admin.27083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36638.log 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27083.log.gz 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34911.log 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33666.log.gz 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29837.log 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36638.log.gz 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40786.log 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34911.log.gz 2024-09-22T03:06:29.206 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29837.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29837.log.gz 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35885.log 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30675.log 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40786.log.gz 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35701.log 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35885.log.gz 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36350.log 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30675.log.gz 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33853.log 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35701.log.gz 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36350.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36350.log.gz 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39569.log 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33853.log.gz 2024-09-22T03:06:29.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33346.log 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39569.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39569.log.gz 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34294.log 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38762.log 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33346.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33346.log.gz 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38631.log 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34294.log.gz 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40788.log 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38762.log.gz 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37205.log 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38631.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38631.log.gz 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29652.log 2024-09-22T03:06:29.208 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40788.log.gz 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35080.log 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37205.log.gz 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41780.log 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29652.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29652.log.gz 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40435.log 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35080.log.gz 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35517.log 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41780.log.gz 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32079.log 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40435.log.gz 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35517.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35517.log.gz 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29380.log 2024-09-22T03:06:29.209 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41487.log 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32079.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32079.log.gz 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38988.log 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29380.log.gz 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40052.log 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41487.log.gz 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30368.log 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38988.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38988.log.gz 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40052.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40052.log.gz 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.41613.log 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31988.log 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30368.log.gz 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29422.log 2024-09-22T03:06:29.210 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41613.log.gz 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40955.log 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31988.log.gz 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35305.log 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29422.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29422.log.gz 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31053.log 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40955.log.gz 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41403.log 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35305.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35305.log.gz 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30319.log 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31053.log.gz 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41403.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41403.log.gz 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28978.log 2024-09-22T03:06:29.211 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30319.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30319.log.gz 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31590.log 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28978.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28978.log.gz 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39964.log 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31590.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31590.log.gz 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.30468.log 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39964.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39964.log.gz 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28834.log 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38589.log 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30468.log.gz 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28834.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28834.log.gz 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33811.log 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31381.log 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38589.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38589.log.gz 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-09-22T03:06:29.212 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33811.log.gz 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30755.log 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31381.log.gz 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39520.log 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41149.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gz 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28733.log 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30755.log.gz 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39520.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39520.log.gz 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29965.log 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28733.log: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.28733.log.gz 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30270.log 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.29965.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34161.log 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29965.log.gz 2024-09-22T03:06:29.213 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31461.log 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30270.log.gz 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34161.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34161.log.gz 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37383.log 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31461.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31461.log.gz 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35431.log 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33200.log 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37383.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37383.log.gz 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40347.log 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35431.log.gz 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37609.log 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33200.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33200.log.gz 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30577.log 2024-09-22T03:06:29.214 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40347.log.gz 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28876.log 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37609.log.gz 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28238.log 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30577.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30577.log.gz 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33915.log 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28876.log.gz 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36865.log 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28238.log.gz 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32456.log 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33915.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.33915.log.gz 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38518.log 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36865.log.gz 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31946.log 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32456.log.gz 2024-09-22T03:06:29.215 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30626.log 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.38518.log.gz 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31946.log: gzip 49.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.31946.log.gz --verbose 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr: -- /var/log/ceph/ceph-client.admin.33715.log 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36987.log 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30626.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30626.log.gz 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40305.log 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33715.log.gz 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28706.log 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36987.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36987.log.gz 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29923.log 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40305.log.gz 2024-09-22T03:06:29.216 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28706.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28706.log.gz 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32121.log 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29923.log.gz 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29506.log 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32416.log 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32121.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.32121.log.gz 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29506.log: gzip -5 --verbose -- 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29506.log.gz 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.35747.log 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32416.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32416.log.gz 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40956.log 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41192.log 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35747.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35747.log.gz 2024-09-22T03:06:29.217 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37517.log 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40956.log.gz 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36729.log 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41192.log.gz 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40477.log 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37517.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37517.log.gz 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33895.log 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36729.log.gz 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34722.log 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.40477.log.gz 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41697.log 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33895.log.gz 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34722.log.gz 2024-09-22T03:06:29.218 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39917.log 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41697.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41697.log.gz 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32575.log 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gz 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5/var/log/ceph/ceph-client.admin.39917.log: --verbose -- /var/log/ceph/ceph-client.admin.30128.log 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39917.log.gz 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32575.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32575.log.gz 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40519.log 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41277.log 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30128.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30128.log.gz 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40519.log: gzip -5 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40519.log.gz 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29146.log 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32505.log 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41277.log.gz 2024-09-22T03:06:29.219 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29749.log 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29146.log.gz 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32505.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32505.log.gz 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38323.log 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29749.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29749.log.gz 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32008.log 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38722.log 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38323.log.gz 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30228.log 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32008.log.gz 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40137.log 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38722.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38722.log.gz 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33506.log 2024-09-22T03:06:29.220 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30228.log.gz 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40137.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40137.log.gz 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37875.log 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31013.log 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33506.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33506.log.gz 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40263.log 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37875.log.gz 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38498.log 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.31013.log.gz 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36397.log 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40263.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40263.log.gz 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30835.log 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38498.log.gz 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33297.log 2024-09-22T03:06:29.221 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36397.log.gz 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38673.log 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.30835.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30835.log.gz 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39254.log 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33297.log.gz 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38365.log 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38673.log.gz 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28792.log 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39254.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39254.log.gz 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41571.log 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38365.log.gz 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29464.log 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28792.log.gz 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34582.log 2024-09-22T03:06:29.222 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41571.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41571.log.gz 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.29464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31639.log 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29464.log.gz 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34582.log: gzip -5 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34582.log.gz 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35932.log 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34245.log 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31639.log.gz 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35932.log: gzip -5 50.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.35932.log.gz /var/log/ceph/ceph-client.admin.39274.log 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34245.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34245.log.gz 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38141.log 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37085.log 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39274.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39274.log.gz 2024-09-22T03:06:29.223 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32296.log 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38141.log.gz 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28976.log 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37085.log.gz 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32296.log: gzip 49.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.32296.log.gz -- 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.32376.log 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40179.log 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28976.log.gz 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32212.log 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32376.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32376.log.gz 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39436.log 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40179.log.gz 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35793.log 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32212.log.gz 2024-09-22T03:06:29.224 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38809.log 2024-09-22T03:06:29.225 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39436.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39436.log.gz 2024-09-22T03:06:29.225 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35793.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.35793.log.gz 2024-09-22T03:06:29.225 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38809.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38809.log.gz 2024-09-22T03:06:29.351 INFO:teuthology.orchestra.run.smithi081.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-22T03:06:29.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-22T03:06:40.391 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-22T03:06:40.440 INFO:teuthology.orchestra.run.smithi081.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-22T03:06:42.632 INFO:teuthology.orchestra.run.smithi081.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-22T03:06:42.634 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-09-22T03:06:42.634 INFO:teuthology.orchestra.run.smithi081.stderr:real 0m13.566s 2024-09-22T03:06:42.635 INFO:teuthology.orchestra.run.smithi081.stderr:user 0m24.257s 2024-09-22T03:06:42.635 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m1.024s 2024-09-22T03:06:43.649 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-22T03:06:43.651 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-22T03:06:43.651 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m14.579s 2024-09-22T03:06:43.651 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m24.975s 2024-09-22T03:06:43.651 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m0.926s 2024-09-22T03:06:43.651 INFO:tasks.ceph:Archiving logs... 2024-09-22T03:06:43.651 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914712/remote/smithi081/log 2024-09-22T03:06:43.652 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T03:06:47.841 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914712/remote/smithi190/log 2024-09-22T03:06:47.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T03:06:51.932 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-22T03:06:51.941 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-22T03:06:51.941 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T03:06:51.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T03:06:52.033 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-22T03:06:52.042 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-22T03:06:52.042 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:06:52.045 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:06:52.060 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:06:52.062 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:06:52.064 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:06:52.064 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-09-22T03:06:52.064 INFO:teuthology.orchestra.run.smithi081.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 -2829ns[ -13us] +/- 47ms 2024-09-22T03:06:52.064 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 +114us[ +104us] +/- 51ms 2024-09-22T03:06:52.065 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -155us[ -165us] +/- 68ms 2024-09-22T03:06:52.065 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:06:52.066 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:06:52.066 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-09-22T03:06:52.066 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 -8582ns[ -26us] +/- 47ms 2024-09-22T03:06:52.066 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 +124us[ +124us] +/- 51ms 2024-09-22T03:06:52.067 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 -166us[ -166us] +/- 68ms 2024-09-22T03:06:52.067 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:06:52.067 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-22T03:06:52.102 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-22T03:06:52.102 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-22T03:06:52.131 DEBUG:teuthology.orchestra.run.smithi081:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-22T03:06:52.203 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-22T03:06:52.270 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:06:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:06:52.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:06:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:06:52.328 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-22T03:06:52.339 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-22T03:06:52.348 INFO:teuthology.task.internal:Duration was 936.137909 seconds 2024-09-22T03:06:52.348 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-22T03:06:52.356 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-22T03:06:52.357 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T03:06:52.359 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T03:06:52.397 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:06:52.408 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:06:52.670 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-22T03:06:52.671 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2024-09-22T03:06:52.671 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T03:06:52.731 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-09-22T03:06:52.732 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T03:06:52.758 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-22T03:06:52.759 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T03:06:52.774 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T03:06:52.825 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-22T03:06:52.826 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T03:06:52.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T03:06:52.966 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-22T03:06:52.976 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-22T03:06:52.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T03:06:53.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T03:06:53.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-22T03:06:53.044 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:06:53.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:06:53.068 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2024-09-22T03:06:53.089 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-09-22T03:06:53.116 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:06:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:06:53.149 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:06:53.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:06:53.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-22T03:06:53.187 INFO:teuthology.task.internal:Transferring archived files... 2024-09-22T03:06:53.187 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914712/remote/smithi081 2024-09-22T03:06:53.188 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T03:06:53.223 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914712/remote/smithi190 2024-09-22T03:06:53.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T03:06:53.258 INFO:teuthology.task.internal:Removing archive directory... 2024-09-22T03:06:53.258 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T03:06:53.260 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T03:06:53.308 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-22T03:06:53.321 INFO:teuthology.task.internal:Not uploading archives. 2024-09-22T03:06:53.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-22T03:06:53.333 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-22T03:06:53.333 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T03:06:53.336 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T03:06:53.348 INFO:teuthology.orchestra.run.smithi081.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 03:06 /home/ubuntu/cephtest 2024-09-22T03:06:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 03:06 /home/ubuntu/cephtest 2024-09-22T03:06:53.365 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-22T03:06:53.378 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-22T03:06:53.404 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} duration: 936.1379091739655 flavor: crimson owner: scheduled_teuthology@teuthology success: true 2024-09-22T03:06:53.404 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:06:53.488 INFO:teuthology.run:pass